[Labs-l] wiki replication stopped since September

[Magnus Manske]

Wikimedia Labs not properly supporting infrastructure for volunteers - sad
but mostly non-critical.

Wikimedia Labs not properly supporting infrastructure for other Wikimedia
divisions, for months apparently - unacceptable.


On Wed Nov 26 2014 at 16:14:54 Dan Andreescu <dandreescu at wikimedia.org>
wrote:

> We (analytics team) run wikimetrics on top of the labs data.  For the last
> couple of months, we've seen missing data (data in prod that's not in labs)
> and a big slowdown on some queries that used to be fast.  I was told about
> the cold indices theory but if it takes this long to warm them up maybe we
> should turn up the thermostat.  Or... however that works in db world :)
>
> Note: we absolutely adore our DBA and labs support folks, we think there
> may just be too much work that needs to get done and these problems are
> definitely affecting our products.  Right now, we have to tell people to
> basically ignore the data for the last couple of months until this get
> sorted out.
>
> On Wed, Nov 26, 2014 at 9:57 AM, Maximilian Doerr <
> maximilian.doerr at gmail.com> wrote:
>
>> Could this in anyway, no matter how remote, be a link to the slow query
>> times enwiki has been spitting out?  Anomie suggests they are cold files in
>> the database, but I don’t understand how that should be happening with
>> indexes in that area.
>>
>> Cyberpower678
>> English Wikipedia Account Creation Team
>> Mailing List Moderator
>>
>>
>>
>> On Nov 26, 2014, at 09:54, Magnus Manske <magnusmanske at googlemail.com>
>> wrote:
>>
>> I already submitted
>>
>> https://phabricator.wikimedia.org/T75980
>>
>> and
>>
>> https://phabricator.wikimedia.org/T75511
>>
>> On Wed Nov 26 2014 at 13:44:39 John <phoenixoverride at gmail.com> wrote:
>>
>>> Actually it looks like things in the databases are completely screwy.
>>> idwiki.labsdb shows zero tables, but enwiki.labsdb "use idwiki_p" shows
>>> what it should. Looks like there may be a bigger issue
>>>
>>> On Wed, Nov 26, 2014 at 7:58 AM, Krinkle <krinklemail at gmail.com> wrote:
>>>
>>>> Seems to be at least 62 days behind on commonswiki
>>>>
>>>> tin.eqiad.wmnet$ sql commonswiki
>>>> rc_id: 142360816
>>>> rc_timestamp: 2014-11-26 12:55:31
>>>>
>>>> tools-login$ sql commonswiki
>>>> rc_id: 136903866
>>>> rc_timestamp: 2014-09-24 12:51:42
>>>>
>>>> See also https://tools.wmflabs.org/unpatrollededitstats/commonswiki/
>>>>
>>>> — Krinkle
>>>>
>>>> On 23 Nov 2014, at 12:56, John <phoenixoverride at gmail.com> wrote:
>>>>
>>>> enwiki's replication lag is at 114256 and growing.
>>>> _______________________________________________
>>>> Labs-l mailing list
>>>> Labs-l at lists.wikimedia.org
>>>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Labs-l mailing list
>>>> Labs-l at lists.wikimedia.org
>>>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>>>
>>>>
>>> _______________________________________________
>>> Labs-l mailing list
>>> Labs-l at lists.wikimedia.org
>>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>>
>> _______________________________________________
>> Labs-l mailing list
>> Labs-l at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>
>>
>>
>> _______________________________________________
>> Labs-l mailing list
>> Labs-l at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>
>>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-l/attachments/20141126/80aa1eb9/attachment.html>