[QA] [Engineering] beta cluster is too busy

[Aaron Schulz]

It seems odd that such a modest batch size with wfWaitForSlaves() could
cause so much lag anywhere. It seems like something is off somewhere.


On Tue, Jul 22, 2014 at 12:54 PM, James Forrester <jforrester at wikimedia.org>
wrote:

> On 22 July 2014 09:24, Antoine Musso <hashar+wmf at free.fr> wrote:
>
>> Hello,
>>
>> Yesterday a patch landed in MediaWiki core that causes the database
>> update script to update a bunch of tables and do a lot of queries on the
>> database.
>>
>> The slave database is thus lagged out by 4+ hours as I write this and
>> that causes a bunch of errors when browsing the beta sites.
>>
>> There is a few more details on:
>>
>>   populateBacklinkNamespace script causing massive slave lag on beta
>>   https://bugzilla.wikimedia.org/show_bug.cgi?id=68349
>>
>> Bryan Davis pointed me to a quite useful link to check the database lag:
>>
>>  http://en.wikipedia.beta.wmflabs.org/w/index.php?maxlag=-1
>>
>>
>> Jenkins is still processing the upgrade of simplewiki which is the
>> largest wiki on beta cluster.  The job output is:
>>
>>
>> https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/label=deployment-bastion-eqiad,wikidb=simplewiki/2742/console
>>
>> Whenever that is completed, the database slave will have to catch up and
>> we will eventually resume normal operations.
>>
>
> ​As an update, the database upgrade finally finished about 20 minutes ago,
> and slave lag is slowly adjusting back to normal. Unfortunately the lag is
> currently nearly four hours long, so this will take a while.
>
> J.
> --
> James D. Forrester
> Product Manager, Editing
> Wikimedia Foundation, Inc.
>
> jforrester at wikimedia.org | @jdforrester
>
> _______________________________________________
> Engineering mailing list
> Engineering at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/engineering
>
>


-- 
-Aaron S
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/qa/attachments/20140722/8f06fd89/attachment.html>