On Nov 6, 2020, at 12:12 PM, Brooke Storm &lt;bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org>> wrote: The ToolsDB service suffered a breakage in replication on 2020-10-27. WMCS has tried to restore replication of data, but that has been unsuccessful so far including doing a dump to rebuild replication without downtime. At this point, we have a new server waiting to become the replica, but to start the replication process, we need to set the database to read-only for a full dump. This could easily take more than an hour. During that entire time, the database will be read-only. We will begin at 1600 UTC and finish when it is done. The database is quite large, but, with it in read-only mode, I hope the backup will not take terribly long. Please see https://phabricator.wikimedia.org/T266587 <https://phabricator.wikimedia.org/T266587> for additional information. Brooke Storm Staff SRE Wikimedia Cloud Services bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org> IRC: bstorm

On Nov 11, 2020, at 8:31 AM, Brooke Storm &lt;bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org>> wrote: Update: ToolsDB remains in read-only mode while the data loads on the replica in order to minimize the possibility that this won’t work. All steps have been successful so far, but the size and heterogeneity of this database service has made every step take large amounts of time for the system to complete. The last setup step is running now and should complete during my day today (US MST). Based on how long this took yesterday to export the data, my worst-case scenario estimate for restoring services is at around 22:30 UTC today. We are tracking work at https://phabricator.wikimedia.org/T266587 <https://phabricator.wikimedia.org/T266587>