[Engineering] The train will resume tomorrow (was Re: All wikis reverted to wmf.8 last night due to T119736)

[Greg Grossmeier]

<quote name="Greg Grossmeier" date="2016-07-12" time="09:24:38 -0700">
> https://phabricator.wikimedia.org/T119736 - "Could not find local user data for {Username}@{wiki}"
> 
> There was an order of magnitude increase in the rate of those errors
> that started on July 7th.
> 
> Investigation and remediation is on-going.

Investigation and remediation is mostly complete[0] and the vast
majority of cases have been addressed. There are still users who will
experience this error for the next ~1 day.[1]

1.28.0-wmf.10 will be branched tomorrow and we will run an abbreviated
train schedule (group0 and group1 on Wednesday, group2 on Thursday).

Thanks to Matt Flaschen and Brad Jorsch (and others like Ori Livneh and
Bryan Davis) for their help.

Sorry for the inconvenience.

Greg


[0] Modulo https://phabricator.wikimedia.org/T140156 which shouldn't
effect auto-creation.

[1] Users who were affected by this already before the fixes in the
code were deployed will still have the issue until the script that fixes
those cases completes running, which takes roughly 1 day. There is a run
of it going now, but we will run it again as we deployed fixes mid-run.

-- 
| Greg Grossmeier            GPG: B2FA 27B1 F7EB D327 6B8E |
| identi.ca: @greg                A18D 1138 8E47 FAC8 1C7D |
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <https://lists.wikimedia.org/pipermail/engineering/attachments/20160712/0e700025/attachment.sig>