The change-over was successful. Thank you for your patience and understanding. ToolsDB should now be fully operational. Please report issues to the #wikimedia-cloud IRC channel. In the course of recovery efforts, the four databases that are not replicated were left behind. This ticket is there to track further recovery efforts on those databases as is desired or possible relating to those four databases listed here: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#ToolsDB_Backups… https://phabricator.wikimedia.org/T216441 Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm(a)wikimedia.org IRC: bstorm_ On Feb 18, 2019, at 9:57 AM, Brooke Storm <bstorm(a)wikimedia.org> wrote: This is an update on the ongoing problems with the toolsdb service. We are preparing to move to a new server, which is now a functioning replica of the toolsdb server. The first step here is to restart the service in read-only mode, and then we will move the DNS. Expect writes to stop working and connections to drop. When we are moved to the new DNS, services that use this database will need to be restarted. This will be happening within the next hour unless it is slowed down by some issues or caution. Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm(a)wikimedia.org IRC: bstorm_ _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce _______________________________________________ Wikimedia Cloud Services mailing list Cloud(a)lists.wikimedia.org (formerly labs-l(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud

Thanks Brooke, everything looks great to me, I really appreciate everybody's work to get this back! Arthur On Mon, Feb 18, 2019 at 2:39 PM Gerard Meijssen <gerard.meijssen(a)gmail.com> wrote: stressful. I thank you for getting this far. I can imagine that you are not done yet but do get your rest. It is deserved. understanding. ToolsDB should now be fully operational. Please report issues to the #wikimedia-cloud IRC channel. replicated were left behind. This ticket is there to track further recovery efforts on those databases as is desired or possible relating to those four databases listed here: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#ToolsDB_Backups… are preparing to move to a new server, which is now a functioning replica of the toolsdb server. The first step here is to restart the service in read-only mode, and then we will move the DNS. Expect writes to stop working and connections to drop. When we are moved to the new DNS, services that use this database will need to be restarted. some issues or caution. labs-announce(a)lists.wikimedia.org) _______________________________________________ Wikimedia Cloud Services mailing list Cloud(a)lists.wikimedia.org (formerly labs-l(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud

Thanks, all looking good! On Tue, Feb 19, 2019 at 2:57 PM Goran Milovanovic < goran.milovanovic_ext(a)wikimedia.de&gt; wrote: _______________________________________________ Wikimedia Cloud Services mailing list Cloud(a)lists.wikimedia.org (formerly labs-l(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud