On Oct 20, 2020, at 1:13 PM, Brooke Storm &lt;bstorm(a)wikimedia.org&gt; wrote: On Tuesday 2020-10-27 at 1600 UTC, ToolsDB, the user database service provided by Toolforge (https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#User_databases <https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#User_databases>) will see some service interruption from restarts and will be set to read-only mode during a scheduled failover. The failover must be done to protect data and minimize downtime while the hypervisor goes down for upgrades. Tools may need to restart in order to reconnect during this time and nothing will be able to write to the database tables until failover is completed. After failover, services should resume as normal, and we will announce when it is done. The process will take some time because we must copy over several databases by hand due to three temporarily unreplicated tables being repaired while things are in read-only mode and the four databases that are always unreplicated being copied (https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#ToolsDB_Backups… <https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#ToolsDB_Backups_and_Replication>). I expect at least half an hour in read-only mode, but things could take longer or shorter time depending on volume of data to copy and issues encountered during the process. Details are on Phabricator at https://phabricator.wikimedia.org/T263679 <https://phabricator.wikimedia.org/T263679> Brooke Storm Staff SRE Wikimedia Cloud Services bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org> IRC: bstorm

Due to recent, unexpected database errors, we are rescheduling this maintenance for this coming Monday at 16:00 UTC. On 10/20/20 3:13 PM, Brooke Storm wrote: