After consulting with several people on several departments, we have scheduled a maintenance window (downtime) for dbstore1002 (alias analytics-store, and s2 *to* s7-analytics-slave) for Thursday 21 July between14:00-15:00 UTC [0].
The downtime is expected to be of only 5-10 minutes, but in case something goes wrong, we will reserve the entire hour. During the maintenance, this host will stop replication from eventlogging and the production shards, kill all ongoing queries and stop responding to new ones. User databases on this host will be unavailable, too.
The recommendation is to use db1047 / analytics-slave / s1-analytics-slave, which also has an up-to-date version of eventlogging and shards s1 and s2 for the following 3 days to avoid service interruption. I see some crons running on dbstore1002, contact me privately if you want to change them and require help.
I will send an update when the maintenance is over to confirm normal work can resume.
On Tue, Jul 19, 2016 at 10:53 AM, Jaime Crespo jcrespo@wikimedia.org wrote:
After consulting with several people on several departments, we have scheduled a maintenance window (downtime) for dbstore1002 (alias analytics-store, and s2 *to* s7-analytics-slave) for Thursday 21 July between14:00-15:00 UTC [0].
[...]
I will send an update when the maintenance is over to confirm normal work can resume.
The maintenance has finished with no known incidents. I have to apologize because despite maintaining the promise of a downtime of only a few minutes, I had to go over the promised time frame of 15:00 UTC due to unrelated, but conflicting network maintenance. As it finally finished at 15:17, I decided to continue past the time rather than reschedule it again.
The current status is that the database is up and available, although for a few hours there will be some lag until it catches up with the eventlogging and s* shard masters and it may be slower than usual. My recommendation is to keep using db1047, when possible, until tomorrow, when everything will be nice and stable.
Yours faithfully,