Wikimedia wikis hosted on the "s3" cluster (pretty much all but the very large wikis, click on the "s3" box in https://noc.wikimedia.org/dbtree/ to get a full list) are currently in read-only mode due to severe replication lag. This problem was apparently caused by a logic issue in the job queue, which should now be fixed, but it will still take at least 2-3 hours for replication to catch up. We apologize for the inconvenience, and are continuing to monitor the situation.
Erik
Στις 11-12-2012, ημέρα Τρι, και ώρα 01:10 -0800, ο/η Erik Moeller έγραψε:
Wikimedia wikis hosted on the "s3" cluster (pretty much all but the very large wikis, click on the "s3" box in https://noc.wikimedia.org/dbtree/ to get a full list) are currently in read-only mode due to severe replication lag. This problem was apparently caused by a logic issue in the job queue, which should now be fixed, but it will still take at least 2-3 hours for replication to catch up. We apologize for the inconvenience, and are continuing to monitor the situation.
Erik
Forgot to follow up here as well: editing was restored at around 11 am UTC, and all databases are back to normal. The job runners have been in operation during this entire time and seem not to cause any db issues now.
Ariel
wikitech-l@lists.wikimedia.org