Hello,
I will be running this schema change on s2 on Monday. Expect delay on s2 on the replicas.
Manuel.
On Wed, Nov 29, 2017 at 1:53 PM, Manuel Arostegui marostegui@wikimedia.org wrote:
Hey Cloud Team,
I am now running this schema changes on s3, for all the wikis (around 900). I have throttled it a bit and it has been running for an hour without any significant delay on the new replicas. labsdb1003 is delayed a bit, but it normally is lately, so I don't think it is related to this change. This should take another 15h or so to finish completely.
Cheers Manuel.
On Wed, Nov 15, 2017 at 6:45 PM, Manuel Arostegui manuel@wikimedia.org wrote:
On Wed, Nov 15, 2017 at 6:39 PM, Bryan Davis bd808@wikimedia.org wrote:
On Wed, Nov 15, 2017 at 9:48 AM, Manuel Arostegui manuel@wikimedia.org wrote:
Hello Cloud Admins!
As part of https://phabricator.wikimedia.org/T174569 we have to alter
some
big tables. One of them is logging, which, for instance, in wikidata takes around
8h.
Which is the shard I am currently working on.
Because of the nature of the change (some columns being added) and ROW
based
replication (what we use in sanitariums) this change needs to be done
with
replication (from sanitarium, or their masters, to the labs servers).
This will obviously generate lag and if not done that way, it will
break
replication till the column is added on the labs hosts, and this is
less
desirable than replication lag.
I am planning to run the alter probably tomorrow or Monday (I will
notify
when I start it) for the sanitarium host in s5, that means that there
will
be lag on the labs servers, for a few hours, on the s5 instance (which
will
also affect s1 and s3 because we are using the same replication thread
for
those shards too - which is a FIXME we have pending).
s2, s4, s6 and s7 will remain unaffected as they have their own
replication
thread.
Should you have any questions, let me know!
Should we send a message to cloud-announce about this, or just be ready to tell people that the lag is a known issue due to production schema changes?
Don't think it is necessary to send an announcement about it, it is just maintenance. I would suggest you just just to point people to that task so they can know when other shards will be done too :-)
Manuel.