I have now had multiple issues in Translate extension that are related to database replication. After doing write to the master, some maintenance tasks kick in and use slave to read the (now stale) data.
I've heard that there is some automatic handling for this, so that editors for example see the latest version of Article after saving it. But apparently there is no such thing, or at least it doesn't kick in automatically.
What should I do? I don't fancy adding parameters to many methods to define whether to use master or slave for data retrieval. Is there method that waits for the used slave to catch up with the state after last write? -Niklas
Niklas Laxström wrote:
I have now had multiple issues in Translate extension that are related to database replication. After doing write to the master, some maintenance tasks kick in and use slave to read the (now stale) data.
The maintenance script need to wait for the slave to catchup the master position. The wfWaitForSlaves() global function is exactly intended for that. A typical use cases is when you make tons of writes to master and dont want to break replication or have the slaves lagging too much. You would split the writes in batches and then wait until slaves catch up.
Have a look at maintenance/runBatchedQuery.php for an easy to read example.
I've heard that there is some automatic handling for this, so that editors for example see the latest version of Article after saving it. But apparently there is no such thing, or at least it doesn't kick in automatically.
We had that issue at one point, I am not sure how it get fixed though. Maybe we fetch the article from the master database to make sure it is up to date after the user hit the save button.
What should I do? I don't fancy adding parameters to many methods to define whether to use master or slave for data retrieval. Is there method that waits for the used slave to catch up with the state after last write? -Niklas
What is your code? :-)
(note I am out for the weekend so catch up with that Monday).
On 16/06/12 14:26, Antoine Musso wrote:
I've heard that there is some automatic handling for this, so that editors for example see the latest version of Article after saving it. But apparently there is no such thing, or at least it doesn't kick in automatically.
We had that issue at one point, I am not sure how it get fixed though. Maybe we fetch the article from the master database to make sure it is up to date after the user hit the save button.
You mean the recent bug 37225 ? (stale data problems have happened several times in the past). It should have fixed with https://gerrit.wikimedia.org/r/#/c/10722/1
On 16 June 2012 15:26, Antoine Musso hashar+wmf@free.fr wrote:
Niklas Laxström wrote:
I have now had multiple issues in Translate extension that are related to database replication. After doing write to the master, some maintenance tasks kick in and use slave to read the (now stale) data.
The maintenance script need to wait for the slave to catchup the master position. The wfWaitForSlaves() global function is exactly intended for that. A typical use cases is when you make tons of writes to master and dont want to break replication or have the slaves lagging too much. You would split the writes in batches and then wait until slaves catch up.
Have a look at maintenance/runBatchedQuery.php for an easy to read example.
The latest case is https://bugzilla.wikimedia.org/37647 - it is not a maintenance script and the problem is not lot of writes, but getting fresh data after write. -Niklas
wikitech-l@lists.wikimedia.org