On Oct 20, 2020, at 1:13 PM, Brooke Storm
<bstorm(a)wikimedia.org> wrote:
On Tuesday 2020-10-27 at 1600 UTC, ToolsDB, the user database service provided by
Toolforge (
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#User_databases
<https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#User_databases>) will
see some service interruption from restarts and will be set to read-only mode during a
scheduled failover. The failover must be done to protect data and minimize downtime while
the hypervisor goes down for upgrades.
Tools may need to restart in order to reconnect during this time and nothing will be able
to write to the database tables until failover is completed.
After failover, services should resume as normal, and we will announce when it is done.
The process will take some time because we must copy over several databases by hand due to
three temporarily unreplicated tables being repaired while things are in read-only mode
and the four databases that are always unreplicated being copied
(
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#ToolsDB_Backups…
<https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#ToolsDB_Backups_and_Replication>).
I expect at least half an hour in read-only mode, but things could take longer or shorter
time depending on volume of data to copy and issues encountered during the process.
Details are on Phabricator at
https://phabricator.wikimedia.org/T263679
<https://phabricator.wikimedia.org/T263679>
Brooke Storm
Staff SRE
Wikimedia Cloud Services
bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org>
IRC: bstorm
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)