[Engineering] 1.27.0-wmf.23 rolled back from Wikipedias

[Gilles Dubuc]

Investigation and likely explanation on
https://phabricator.wikimedia.org/T134553

tl;dr: the source of the spike seems to be in the instrumentation and the
newer higher values should be closer to reality.

It is safe to roll out 1.27-wmf.23 ASAP. Releng, please let me know when
that will be, so that I can keep an eye on things to confirm my findings.

Sorry about the inconvenience!

On Fri, May 6, 2016 at 5:14 PM, Benoît Evellin (Trizek) <
bevellin at wikimedia.org> wrote:

> I'm currently writing the next Tech News issue. Is that bug will delay the
> next MediaWiki deployment?
>
> Thanks,
> Benoît
>
> On Fri, May 6, 2016 at 8:43 AM, Ori Livneh <ori at wikimedia.org> wrote:
>
>> The roll-out of 1.27.0-wmf.23 to all Wikipedias coincided with a
>> significant (~100-150ms) increase in first paint time for Wikimedia page
>> views. First paint time is the time it takes the user to see anything on
>> the screen after clicking on a link. Keeping it low is crucial to good user
>> experience, and this is a severe regression, so I rolled back wmf.23, in
>> coordination with Chad and Tyler from Release Engineering.
>>
>> In the graph below, the green line shows median first paint time in
>> milliseconds over the course of the past 24 hours. The blue line shows what
>> the value was that time the day before. You can see they diverge shortly
>> after 19:00 UTC and converge around 4:20 UTC.
>>
>> <removed>
>> ​
>> I have not investigated any further than that yet, but the train is
>> blocked on isolating and fixing this regression (or disproving it, if it is
>> a bug in the instrumentation). This is tracked in Phabricator as
>> https://phabricator.wikimedia.org/T134553. You are encouraged to help
>> out. :)
>>
>> Ori
>>
>> _______________________________________________
>> Engineering mailing list
>> Engineering at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/engineering
>>
>>
>
>
> --
> Benoît Evellin (Trizek)
> Community Liaison
> Wikimedia Foundation
>
> _______________________________________________
> Engineering mailing list
> Engineering at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/engineering
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/engineering/attachments/20160509/e4fa1ffe/attachment.html>