Hello Cloud Admins!
As part of https://phabricator.wikimedia.org/T174569 we have to alter some
big tables.
One of them is logging, which, for instance, in wikidata takes around 8h.
Which is the shard I am currently working on.
Because of the nature of the change (some columns being added) and ROW
based replication (what we use in sanitariums) this change needs to be done
with replication (from sanitarium, or their masters, to the labs servers).
This will obviously generate lag and if not done that way, it will break
replication till the column is added on the labs hosts, and this is less
desirable than replication lag.
I am planning to run the alter probably tomorrow or Monday (I will notify
when I start it) for the sanitarium host in s5, that means that there will
be lag on the labs servers, for a few hours, on the s5 instance (which will
also affect s1 and s3 because we are using the same replication thread for
those shards too - which is a FIXME we have pending).
s2, s4, s6 and s7 will remain unaffected as they have their own replication
thread.
Should you have any questions, let me know!
Thanks
Manuel.
2017-11-29 20:00:02,315 INFO force is enabled
2017-11-29 20:00:02,357 INFO removing misc-project-backup
2017-11-29 20:00:02,456 INFO removing misc-project-backup
2017-11-29 20:00:03,196 INFO creating misc-project-backup at 2T
2017-11-29 20:00:03,960 INFO force is enabled
2017-11-29 20:00:04,009 INFO removing misc-snap
2017-11-29 20:00:04,060 INFO removing misc-snap
2017-11-29 20:00:04,532 INFO creating misc-snap at 1T
Most people attending meeting with ops or SOS will be familiar with it, but
on this quarter we will have some potentially breaking changes on the
mediawiki databases:
* T177208: We have scheduled for the 9th January moving wikidata from "s5"
to "s8" to provide dedicated resources to it. dewiki will stay at "s5".
Config is already prepared:
https://noc.wikimedia.org/conf/highlight.php?file=db-eqiad.php but it will
not be effective until that day, when we will have a read-only period to
perform the split. Most of mediawiki should be ok with it, but there could
be scripts hardcoding wikidata is on s5 (specially on toolforge scripts- we
should coordinate an announcement with cloud there)
* T178359: We now have production multi-instance hosts (meaning multiple
instances per physical host). That means that mysql will not be available
only on port 3306 (default), you may have to indicate a specific socket
(mysql --skip-ssl --socket=/run/mysqld/mysqld.s1.sock) or port (-P3311).
You can see the list of hosts on tendril or grafana.
Having this on an email can become handy in case of emergency.
Cheers,
--
Jaime Crespo
<http://wikimedia.org>
2017-11-28 20:00:03,179 INFO force is enabled
2017-11-28 20:00:03,228 INFO removing tools-project-backup
2017-11-28 20:00:03,328 INFO removing tools-project-backup
2017-11-28 20:00:03,978 INFO creating tools-project-backup at 2T
2017-11-28 20:00:04,847 INFO force is enabled
2017-11-28 20:00:04,879 INFO removing tools-snap
2017-11-28 20:00:04,913 INFO removing tools-snap
2017-11-28 20:00:06,096 INFO creating tools-snap at 1T
I tried to whip something up. I don't think it is trivially easy or
horribly hard, but I'd like y'all to take a quick look before I give
it to Liz so she can start handing it out to candidates. There's one
sneaky question in there that a candidate may figure out an answer to,
but its mostly to see if they can ask good follow up questions. The
others should all be not too hard to find in the support documents I
linked (I hope).
https://etherpad.wikimedia.org/p/WMCS-techsupport-task
Bryan
--
Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org>
[[m:User:BDavis_(WMF)]] Manager, Cloud Services Boise, ID USA
irc: bd808 v:415.839.6885 x6855
2017-11-22 20:00:02,419 INFO force is enabled
2017-11-22 20:00:02,463 INFO removing misc-project-backup
2017-11-22 20:00:02,563 INFO removing misc-project-backup
2017-11-22 20:00:02,944 INFO creating misc-project-backup at 2T
2017-11-22 20:00:03,810 INFO force is enabled
2017-11-22 20:00:03,828 INFO removing misc-snap
2017-11-22 20:00:03,886 INFO removing misc-snap
2017-11-22 20:00:04,802 INFO creating misc-snap at 1T
2017-11-21 20:00:02,259 INFO force is enabled
2017-11-21 20:00:02,279 INFO removing tools-project-backup
2017-11-21 20:00:02,314 INFO removing tools-project-backup
2017-11-21 20:00:02,761 INFO creating tools-project-backup at 2T
2017-11-21 20:00:03,508 INFO force is enabled
2017-11-21 20:00:03,552 INFO removing tools-snap
2017-11-21 20:00:03,598 INFO removing tools-snap
2017-11-21 20:00:05,243 INFO creating tools-snap at 1T
2017-11-15 20:00:02,267 INFO force is enabled
2017-11-15 20:00:02,300 INFO removing misc-project-backup
2017-11-15 20:00:02,391 INFO removing misc-project-backup
2017-11-15 20:00:02,781 INFO creating misc-project-backup at 2T
2017-11-15 20:00:03,627 INFO force is enabled
2017-11-15 20:00:03,664 INFO removing misc-snap
2017-11-15 20:00:03,736 INFO removing misc-snap
2017-11-15 20:00:04,012 INFO creating misc-snap at 1T