This will be happening in around 10 minutes. ToolsDB will be read-only until we can get a consistent dump to rebuild replication.

Brooke Storm
Staff SRE
Wikimedia Cloud Services
bstorm@wikimedia.org
IRC: bstorm

On Nov 6, 2020, at 12:12 PM, Brooke Storm <bstorm@wikimedia.org> wrote:

The ToolsDB service suffered a breakage in replication on 2020-10-27. WMCS has tried to restore replication of data, but that has been unsuccessful so far including doing a dump to rebuild replication without downtime.
At this point, we have a new server waiting to become the replica, but to start the replication process, we need to set the database to read-only for a full dump. This could easily take more than an hour. During that entire time, the database will be read-only.

We will begin at 1600 UTC and finish when it is done. The database is quite large, but, with it in read-only mode, I hope the backup will not take terribly long.

Please see https://phabricator.wikimedia.org/T266587 for additional information. 

Brooke Storm
Staff SRE
Wikimedia Cloud Services
IRC: bstorm