I note that MNT-1225, which was originally projected to take over 24 hours,
is now closing in on 72 hours and there doesn't seem to be any meaningful
way to project a completion time. Nothing we can do about that now, of
course.
But, I am wondering whether it was necessary to run these updates on *both*
replicas of s1 (s1-sql-rr and s1-sql-user) at the same time? This is really
a naive question, as I don't know enough about mysql administration to even
guess at the answer. Perhaps someone more knowledgeable can enlighten us.
(I do observe, however, that WMF seems to have managed to update all of its
database slaves in some kind of sequential fashion that didn't impact access
to enwiki.)
If, in fact, there is no technical requirement for updating both replicas at
the same time, I would suggest that the next time a situation like this
arises, it would make more sense to do the updates sequentially so that
users (both toolserver users and tool users) are not deprived of access to
this resource for such a long time.
Russ