[Engineering] Reverted to 1.28.0-wmf.8 once more.

[Mukunda Modell]

Final update:

Anomie's theory was spot on. 1.29.0-wmf.9 has been deployed to all wikis
for ~7 hours and appears to be stable.

I've filed a follow-up task for the root cause. It will very likely bite us
in the future if not properly addressed. Further discussion is on
phabricator: https://phabricator.wikimedia.org/T156541

Thanks to everyone who pitched in on this and especially to Anomie for
getting to the bottom of it.

Have a great weekend everyone!

- Mukunda

---
Mukunda Modell
Wikimedia Release Engineering


On Fri, Jan 27, 2017 at 12:47 PM, Mukunda Modell <mmodell at wikimedia.org>
wrote:

> Status update:
>
> Anomie has a solid theory about the cause of
> https://phabricator.wikimedia.org/T156364 and we are going ahead with
> deploying wmf.9 today ASAP.  Assuming everything goes as planned then this
> will be resolved shortly.
>
> On Fri, Jan 27, 2017 at 11:02 AM, Mukunda Modell <mmodell at wikimedia.org>
> wrote:
>
>> Correction: It looks like I only reverted group2 to wmf.8
>>
>> I'm not sure whether to go ahead and revert all wikis or leave it like it
>> is for now.
>>
>> On Fri, Jan 27, 2017 at 10:24 AM, Mukunda Modell <mmodell at wikimedia.org>
>> wrote:
>>
>>> The deployment of MediaWiki 1.29.0-wmf.9 is currently blocked and all
>>> wikis have been reverted to wmf.8.  This latest revert is due to excessive
>>> warnings which appeared in logs immediately after deploying to group2. The
>>> warnings all read as follows:
>>>
>>> Warning: Empty regular expression in /srv/mediawiki/php-1.29.0-wmf.
>>> 9/includes/parser/DateFormatter.php on line 200
>>>
>>> This issue is tracked in https://phabricator.wikimedia.org/T156364 and
>>> currently blocks further deployments.
>>>
>>> More background:
>>>
>>> A similar flood of log spam occurred after deploying the branch to
>>> group1, however, the problem subsided after a while and did not reappear
>>> until Thursday's deployment.
>>>
>>> There have been recent changes to DateFormatter.php, however, the
>>> changes do not appear to be the direct culprit. In fact, after rolling back
>>> the changes, a different message appears in the log, however, with less
>>> frequency than the line 200 error.
>>>
>>> Several engineers have looked at this issue but so far we have not
>>> isolated the root cause. Additional eyes on this issue would be welcomed
>>> and further discussion should be directed to Phabricator. In the interest
>>> of keeping everyone in the loop, I will follow up in this thread if/when
>>> there is a resolution.
>>>
>>> Regards,
>>> Mukunda
>>>
>>> ---
>>> Mukunda Modell
>>> Wikimedia Release Engineering
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/engineering/attachments/20170127/44050072/attachment-0001.html>