[Labs-l] Wikitech maintenance and CI downtime tomorrow, 2015-02-12 16:00 UTC

[Andrew Bogott]

Alex and I have spent much of the day hunting down bugs, and the obvious 
ones are now fixed.  The only issue that I'm still seeing is that if 
you're a member of every single project, logins are very slow... 
fortunately I am probably the only user bit by this :)

So, now:  please open a phab task if you encounter bad behavior on 
wikitech, and assign it to me or direct my attention to it via email or IRC.

Many thanks to Alex Monk for all his code review, and thanks to everyone 
for your patience with the downtime.

-Andrew


On 2/12/16 12:11 PM, Andrew Bogott wrote:
> Update:
>
> This migration is complete.  There are a bunch of bugs with labs 
> features in wikitech which I'm now hunting down one by one.  In the 
> meantime, CI should be working again, and most common wikitech actions 
> should mostly work.
>
> I'll send another 'all clear' once I've stamped out some of the more 
> obvious bugs.  In the meantime maybe hold off on filing phab tickets 
> since most likely we already know :)
>
> -Andrew
>
>
> On 2/11/16 9:54 AM, Andrew Bogott wrote:
>> Executive summary:
>>
>>     CI tests won't run for a while tomorrow.  Also you won't be able 
>> to log into Wikitech.  This will begin at 16:00 UTC (8:00 AM in 
>> California) and may take a couple of hours.
>>
>> Full story:
>>
>>     As part of a long-overdue tech-debt payment[0], I'll be migrating 
>> all of the Labs project, membership and role data from ldap into 
>> mysql[1].  Keystone (the Labs Openstack authentication layer) will be 
>> shut off briefly, and subsequently will have incomplete data while 
>> things are gradually copied over from Ldap.
>>     At the same time, Alex will be rolling out a bunch of 
>> OpenStackManager patches[2] to cope with the new reality.  I've 
>> tested these quite a lot, but no doubt there will be unexpected 
>> issues with such a big refactor.
>>
>>     Existing, running labs and tools sessions should not be 
>> affected.  New instance creation will be disabled for part of the 
>> window, which will prevent CI from starting new tests. Wikitech login 
>> will be disabled, and users will have to login afresh after logins 
>> are restored.
>>
>>     There is, unfortunately, no user-facing improvement associated 
>> with this update.  If all goes perfectly, Wikitech will be restored 
>> to its original state but be slightly slower. If, after the update, 
>> anyone encounters new Wikitech issues (and I emphasize the NEW), 
>> please open a phab ticket and inform me immediately.
>>
>>     As always, in the worst case scenario you can refer to our doc 
>> site of last resort, https://wikitech-static.wikimedia.org
>>
>>
>> -Andrew
>>
>>
>> [0] https://phabricator.wikimedia.org/T115029
>> [1] https://wikitech.wikimedia.org/wiki/Labs_keystone_roles
>> [2] The patchset begins with https://gerrit.wikimedia.org/r/#/c/252615/
>