Asher Feldman wrote:
I've temporarily commented out db36 from db.php on
the cluster.
This is a flaw in the how the client-side use of maxlag interacts with our
schema migration process - we run migrations on slaves one by one in an
automated fashion, only moving to the next after replication lag catches
up. Mediawiki takes care of not sending queries to the lagged slave that
is under migration. Meanwhile, maxlag always reports the value of the most
lagged slave. Not a new issue, but this particular alter table on enwiki
is likely the most time intensive ever run at wmf. It's slightly
ridiculous.
For this one alter, I can stop the migration script and run each statement
by hand, pulling and re-adding db's one by one along the way, but this
isn't a sustainable process. Perhaps we can add a migration flag to
mediawiki, which if enabled, changes the behavior of maxlag and
wfWaitForSlaves() to ignore one highly lagged slave so long as others are
available without lag.
Thank you! My scripts are running fine again. :-)
Some people have suggested that the current API behavior is intentional.
That is, that having different servers return the same error code is better
than having different servers return an error code or not. I think this is
flawed logic due to the problems that it presents (scripts unable to get
past the error code), but it's definitely something that needs investigation
for the future.
MZMcBride