<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><span style="font-family:arial">On 22 July 2014 09:24, Antoine Musso </span><span dir="ltr" style="font-family:arial"><<a href="mailto:hashar+wmf@free.fr" target="_blank">hashar+wmf@free.fr</a>></span><span style="font-family:arial"> wrote:</span><br>
</div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hello,<br>
<br>
Yesterday a patch landed in MediaWiki core that causes the database<br>
update script to update a bunch of tables and do a lot of queries on the<br>
database.<br>
<br>
The slave database is thus lagged out by 4+ hours as I write this and<br>
that causes a bunch of errors when browsing the beta sites.<br>
<br>
There is a few more details on:<br>
<br>
populateBacklinkNamespace script causing massive slave lag on beta<br>
<a href="https://bugzilla.wikimedia.org/show_bug.cgi?id=68349" target="_blank">https://bugzilla.wikimedia.org/show_bug.cgi?id=68349</a><br>
<br>
Bryan Davis pointed me to a quite useful link to check the database lag:<br>
<br>
<a href="http://en.wikipedia.beta.wmflabs.org/w/index.php?maxlag=-1" target="_blank">http://en.wikipedia.beta.wmflabs.org/w/index.php?maxlag=-1</a><br>
<br>
<br>
Jenkins is still processing the upgrade of simplewiki which is the<br>
largest wiki on beta cluster. The job output is:<br>
<br>
<a href="https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/label=deployment-bastion-eqiad,wikidb=simplewiki/2742/console" target="_blank">https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/label=deployment-bastion-eqiad,wikidb=simplewiki/2742/console</a><br>
<br>
Whenever that is completed, the database slave will have to catch up and<br>
we will eventually resume normal operations.<br></blockquote><div><br></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">As an update, the database upgrade finally finished about 20 minutes ago, and slave lag is slowly adjusting back to normal. Unfortunately the lag is currently nearly four hours long, so this will take a while.</div>
</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">J.</div></div>-- <br><div dir="ltr">James D. Forrester<br>
Product Manager, Editing<br>Wikimedia Foundation, Inc.<br><br><a href="mailto:jforrester@wikimedia.org" target="_blank">jforrester@wikimedia.org</a> | @jdforrester</div>
</div></div>