Executive summary:
CI tests won't run for a while tomorrow. Also you won't be able to log into Wikitech. This will begin at 16:00 UTC (8:00 AM in California) and may take a couple of hours.
Full story:
As part of a long-overdue tech-debt payment[0], I'll be migrating all of the Labs project, membership and role data from ldap into mysql[1]. Keystone (the Labs Openstack authentication layer) will be shut off briefly, and subsequently will have incomplete data while things are gradually copied over from Ldap. At the same time, Alex will be rolling out a bunch of OpenStackManager patches[2] to cope with the new reality. I've tested these quite a lot, but no doubt there will be unexpected issues with such a big refactor.
Existing, running labs and tools sessions should not be affected. New instance creation will be disabled for part of the window, which will prevent CI from starting new tests. Wikitech login will be disabled, and users will have to login afresh after logins are restored.
There is, unfortunately, no user-facing improvement associated with this update. If all goes perfectly, Wikitech will be restored to its original state but be slightly slower. If, after the update, anyone encounters new Wikitech issues (and I emphasize the NEW), please open a phab ticket and inform me immediately.
As always, in the worst case scenario you can refer to our doc site of last resort, https://wikitech-static.wikimedia.org
-Andrew
[0] https://phabricator.wikimedia.org/T115029 [1] https://wikitech.wikimedia.org/wiki/Labs_keystone_roles [2] The patchset begins with https://gerrit.wikimedia.org/r/#/c/252615/
If all goes perfectly, Wikitech will be restored to its original state
but be slightly slower.
Could you explain this a bit more? Is this due to the new role data storage backend? How much of a slowdown should we expect?
Thanks, Negative24
On Thu, Feb 11, 2016 at 8:54 AM Andrew Bogott abogott@wikimedia.org wrote:
Executive summary:
CI tests won't run for a while tomorrow. Also you won't be able tolog into Wikitech. This will begin at 16:00 UTC (8:00 AM in California) and may take a couple of hours.
Full story:
As part of a long-overdue tech-debt payment[0], I'll be migratingall of the Labs project, membership and role data from ldap into mysql[1]. Keystone (the Labs Openstack authentication layer) will be shut off briefly, and subsequently will have incomplete data while things are gradually copied over from Ldap. At the same time, Alex will be rolling out a bunch of OpenStackManager patches[2] to cope with the new reality. I've tested these quite a lot, but no doubt there will be unexpected issues with such a big refactor.
Existing, running labs and tools sessions should not be affected.New instance creation will be disabled for part of the window, which will prevent CI from starting new tests. Wikitech login will be disabled, and users will have to login afresh after logins are restored.
There is, unfortunately, no user-facing improvement associated withthis update. If all goes perfectly, Wikitech will be restored to its original state but be slightly slower. If, after the update, anyone encounters new Wikitech issues (and I emphasize the NEW), please open a phab ticket and inform me immediately.
As always, in the worst case scenario you can refer to our doc siteof last resort, https://wikitech-static.wikimedia.org
-Andrew
[0] https://phabricator.wikimedia.org/T115029 [1] https://wikitech.wikimedia.org/wiki/Labs_keystone_roles [2] The patchset begins with https://gerrit.wikimedia.org/r/#/c/252615/
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Update:
This migration is complete. There are a bunch of bugs with labs features in wikitech which I'm now hunting down one by one. In the meantime, CI should be working again, and most common wikitech actions should mostly work.
I'll send another 'all clear' once I've stamped out some of the more obvious bugs. In the meantime maybe hold off on filing phab tickets since most likely we already know :)
-Andrew
On 2/11/16 9:54 AM, Andrew Bogott wrote:
Executive summary:
CI tests won't run for a while tomorrow. Also you won't be ableto log into Wikitech. This will begin at 16:00 UTC (8:00 AM in California) and may take a couple of hours.
Full story:
As part of a long-overdue tech-debt payment[0], I'll be migratingall of the Labs project, membership and role data from ldap into mysql[1]. Keystone (the Labs Openstack authentication layer) will be shut off briefly, and subsequently will have incomplete data while things are gradually copied over from Ldap. At the same time, Alex will be rolling out a bunch of OpenStackManager patches[2] to cope with the new reality. I've tested these quite a lot, but no doubt there will be unexpected issues with such a big refactor.
Existing, running labs and tools sessions should not be affected.New instance creation will be disabled for part of the window, which will prevent CI from starting new tests. Wikitech login will be disabled, and users will have to login afresh after logins are restored.
There is, unfortunately, no user-facing improvement associatedwith this update. If all goes perfectly, Wikitech will be restored to its original state but be slightly slower. If, after the update, anyone encounters new Wikitech issues (and I emphasize the NEW), please open a phab ticket and inform me immediately.
As always, in the worst case scenario you can refer to our docsite of last resort, https://wikitech-static.wikimedia.org
-Andrew
[0] https://phabricator.wikimedia.org/T115029 [1] https://wikitech.wikimedia.org/wiki/Labs_keystone_roles [2] The patchset begins with https://gerrit.wikimedia.org/r/#/c/252615/
Alex and I have spent much of the day hunting down bugs, and the obvious ones are now fixed. The only issue that I'm still seeing is that if you're a member of every single project, logins are very slow... fortunately I am probably the only user bit by this :)
So, now: please open a phab task if you encounter bad behavior on wikitech, and assign it to me or direct my attention to it via email or IRC.
Many thanks to Alex Monk for all his code review, and thanks to everyone for your patience with the downtime.
-Andrew
On 2/12/16 12:11 PM, Andrew Bogott wrote:
Update:
This migration is complete. There are a bunch of bugs with labs features in wikitech which I'm now hunting down one by one. In the meantime, CI should be working again, and most common wikitech actions should mostly work.
I'll send another 'all clear' once I've stamped out some of the more obvious bugs. In the meantime maybe hold off on filing phab tickets since most likely we already know :)
-Andrew
On 2/11/16 9:54 AM, Andrew Bogott wrote:
Executive summary:
CI tests won't run for a while tomorrow. Also you won't be ableto log into Wikitech. This will begin at 16:00 UTC (8:00 AM in California) and may take a couple of hours.
Full story:
As part of a long-overdue tech-debt payment[0], I'll be migratingall of the Labs project, membership and role data from ldap into mysql[1]. Keystone (the Labs Openstack authentication layer) will be shut off briefly, and subsequently will have incomplete data while things are gradually copied over from Ldap. At the same time, Alex will be rolling out a bunch of OpenStackManager patches[2] to cope with the new reality. I've tested these quite a lot, but no doubt there will be unexpected issues with such a big refactor.
Existing, running labs and tools sessions should not beaffected. New instance creation will be disabled for part of the window, which will prevent CI from starting new tests. Wikitech login will be disabled, and users will have to login afresh after logins are restored.
There is, unfortunately, no user-facing improvement associatedwith this update. If all goes perfectly, Wikitech will be restored to its original state but be slightly slower. If, after the update, anyone encounters new Wikitech issues (and I emphasize the NEW), please open a phab ticket and inform me immediately.
As always, in the worst case scenario you can refer to our docsite of last resort, https://wikitech-static.wikimedia.org
-Andrew
[0] https://phabricator.wikimedia.org/T115029 [1] https://wikitech.wikimedia.org/wiki/Labs_keystone_roles [2] The patchset begins with https://gerrit.wikimedia.org/r/#/c/252615/
wikitech-l@lists.wikimedia.org