This maintenance is turning out to have several unexpected blockers, so we are postponing it until further notice. Keep an eye out for future announcements about related database downtime or read-only intervals.

On 10/27/20 11:05 AM, Andrew Bogott wrote:

Due to recent, unexpected database errors, we are rescheduling this maintenance for this coming Monday at 16:00 UTC.


On 10/20/20 3:13 PM, Brooke Storm wrote:
On Tuesday 2020-10-27 at 1600 UTC, ToolsDB, the user database service provided by Toolforge (https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#User_databases) will see some service interruption from restarts and will be set to read-only mode during a scheduled failover. The failover must be done to protect data and minimize downtime while the hypervisor goes down for upgrades.

Tools may need to restart in order to reconnect during this time and nothing will be able to write to the database tables until failover is completed.

After failover, services should resume as normal, and we will announce when it is done. The process will take some time because we must copy over several databases by hand due to three temporarily unreplicated tables being repaired while things are in read-only mode and the four databases that are always unreplicated being copied (https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#ToolsDB_Backups_and_Replication). I expect at least half an hour in read-only mode, but things could take longer or shorter time depending on volume of data to copy and issues encountered during the process.

Details are on Phabricator at https://phabricator.wikimedia.org/T263679 

Brooke Storm
Staff SRE
Wikimedia Cloud Services
IRC: bstorm


_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce