<div dir="ltr">It seems odd that such a modest batch size with wfWaitForSlaves() could cause so much lag anywhere. It seems like something is off somewhere.</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Jul 22, 2014 at 12:54 PM, James Forrester <span dir="ltr"><<a href="mailto:jforrester@wikimedia.org" target="_blank">jforrester@wikimedia.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class=""><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><span style="font-family:arial">On 22 July 2014 09:24, Antoine Musso </span><span dir="ltr" style="font-family:arial"><<a href="mailto:hashar+wmf@free.fr" target="_blank">hashar+wmf@free.fr</a>></span><span style="font-family:arial"> wrote:</span><br>
</div></div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class="">Hello,<br>
<br>
Yesterday a patch landed in MediaWiki core that causes the database<br>
update script to update a bunch of tables and do a lot of queries on the<br>
database.<br>
<br>
The slave database is thus lagged out by 4+ hours as I write this and<br>
that causes a bunch of errors when browsing the beta sites.<br>
<br>
There is a few more details on:<br>
<br>
populateBacklinkNamespace script causing massive slave lag on beta<br>
<a href="https://bugzilla.wikimedia.org/show_bug.cgi?id=68349" target="_blank">https://bugzilla.wikimedia.org/show_bug.cgi?id=68349</a><br>
<br>
Bryan Davis pointed me to a quite useful link to check the database lag:<br>
<br>
<a href="http://en.wikipedia.beta.wmflabs.org/w/index.php?maxlag=-1" target="_blank">http://en.wikipedia.beta.wmflabs.org/w/index.php?maxlag=-1</a><br>
<br>
<br>
Jenkins is still processing the upgrade of simplewiki which is the<br>
largest wiki on beta cluster. The job output is:<br>
<br>
<a href="https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/label=deployment-bastion-eqiad,wikidb=simplewiki/2742/console" target="_blank">https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/label=deployment-bastion-eqiad,wikidb=simplewiki/2742/console</a><br>
<br></div><div class="">
Whenever that is completed, the database slave will have to catch up and<br>
we will eventually resume normal operations.<br></div></blockquote><div><br></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">As an update, the database upgrade finally finished about 20 minutes ago, and slave lag is slowly adjusting back to normal. Unfortunately the lag is currently nearly four hours long, so this will take a while.</div>
<span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">
J.</div></font></span></div><span class="HOEnZb"><font color="#888888">-- <br><div dir="ltr">James D. Forrester<br>
Product Manager, Editing<br>Wikimedia Foundation, Inc.<br><br><a href="mailto:jforrester@wikimedia.org" target="_blank">jforrester@wikimedia.org</a> | @jdforrester</div>
</font></span></div></div>
<br>_______________________________________________<br>
Engineering mailing list<br>
<a href="mailto:Engineering@lists.wikimedia.org">Engineering@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/engineering" target="_blank">https://lists.wikimedia.org/mailman/listinfo/engineering</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br>-Aaron S<br>
</div>