November 2017 - Cloud-admin - lists.wikimedia.org

Upcoming: Delay in new wiki replicas on s5
by Manuel Arostegui 17 Jan '18

17 Jan '18

Hello Cloud Admins! As part of https://phabricator.wikimedia.org/T174569 we have to alter some big tables. One of them is logging, which, for instance, in wikidata takes around 8h. Which is the shard I am currently working on. Because of the nature of the change (some columns being added) and ROW based replication (what we use in sanitariums) this change needs to be done with replication (from sanitarium, or their masters, to the labs servers). This will obviously generate lag and if not done that way, it will break replication till the column is added on the labs hosts, and this is less desirable than replication lag. I am planning to run the alter probably tomorrow or Monday (I will notify when I start it) for the sanitarium host in s5, that means that there will be lag on the labs servers, for a few hours, on the s5 instance (which will also affect s1 and s3 because we are using the same replication thread for those shards too - which is a FIXME we have pending). s2, s4, s6 and s7 will remain unaffected as they have their own replication thread. Should you have any questions, let me know! Thanks Manuel.

3 9

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 30 Nov '17

30 Nov '17

2017-11-29 20:00:02,315 INFO force is enabled 2017-11-29 20:00:02,357 INFO removing misc-project-backup 2017-11-29 20:00:02,456 INFO removing misc-project-backup 2017-11-29 20:00:03,196 INFO creating misc-project-backup at 2T 2017-11-29 20:00:03,960 INFO force is enabled 2017-11-29 20:00:04,009 INFO removing misc-snap 2017-11-29 20:00:04,060 INFO removing misc-snap 2017-11-29 20:00:04,532 INFO creating misc-snap at 1T

1 0

Changes on database infrastructure that could break hardcoded utilities (s8, multiinstance)
by Jaime Crespo 29 Nov '17

29 Nov '17

Most people attending meeting with ops or SOS will be familiar with it, but on this quarter we will have some potentially breaking changes on the mediawiki databases: * T177208: We have scheduled for the 9th January moving wikidata from "s5" to "s8" to provide dedicated resources to it. dewiki will stay at "s5". Config is already prepared: https://noc.wikimedia.org/conf/highlight.php?file=db-eqiad.php but it will not be effective until that day, when we will have a read-only period to perform the split. Most of mediawiki should be ok with it, but there could be scripts hardcoding wikidata is on s5 (specially on toolforge scripts- we should coordinate an announcement with cloud there) * T178359: We now have production multi-instance hosts (meaning multiple instances per physical host). That means that mysql will not be available only on port 3306 (default), you may have to indicate a specific socket (mysql --skip-ssl --socket=/run/mysqld/mysqld.s1.sock) or port (-P3311). You can see the list of hosts on tendril or grafana. Having this on an email can become handy in case of emergency. Cheers, -- Jaime Crespo <http://wikimedia.org>

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 29 Nov '17

29 Nov '17

2017-11-28 20:00:03,179 INFO force is enabled 2017-11-28 20:00:03,228 INFO removing tools-project-backup 2017-11-28 20:00:03,328 INFO removing tools-project-backup 2017-11-28 20:00:03,978 INFO creating tools-project-backup at 2T 2017-11-28 20:00:04,847 INFO force is enabled 2017-11-28 20:00:04,879 INFO removing tools-snap 2017-11-28 20:00:04,913 INFO removing tools-snap 2017-11-28 20:00:06,096 INFO creating tools-snap at 1T

1 0

Tech support interview task
by Bryan Davis 28 Nov '17

28 Nov '17

I tried to whip something up. I don't think it is trivially easy or horribly hard, but I'd like y'all to take a quick look before I give it to Liz so she can start handing it out to candidates. There's one sneaky question in there that a candidate may figure out an answer to, but its mostly to see if they can ask good follow up questions. The others should all be not too hard to find in the support documents I linked (I hope). https://etherpad.wikimedia.org/p/WMCS-techsupport-task Bryan -- Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org> [[m:User:BDavis_(WMF)]] Manager, Cloud Services Boise, ID USA irc: bd808 v:415.839.6885 x6855

1 0

clinic notes for 2017-11-21 through -28
by Andrew Bogott 28 Nov '17

28 Nov '17

not much! It was a short week and IRC was quiet. * That anti-Der-Hexer troll appeared briefly on Thanksgiving but I delegated the cleanup to Nick since he seemed to be working then. * Fulfilled quota and project requests: ** https://phabricator.wikimedia.org/T180178 ( Traffic quota change) ** https://phabricator.wikimedia.org/T180347 (wikidata-primary-sources) ** https://phabricator.wikimedia.org/T180421 (partner-metrics) ** https://phabricator.wikimedia.org/T180662 (qna) * Advised Matanya re: new project creation for nextcloud * Puzzled over https://phabricator.wikimedia.org/T178661

1 0

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 23 Nov '17

23 Nov '17

2017-11-22 20:00:02,419 INFO force is enabled 2017-11-22 20:00:02,463 INFO removing misc-project-backup 2017-11-22 20:00:02,563 INFO removing misc-project-backup 2017-11-22 20:00:02,944 INFO creating misc-project-backup at 2T 2017-11-22 20:00:03,810 INFO force is enabled 2017-11-22 20:00:03,828 INFO removing misc-snap 2017-11-22 20:00:03,886 INFO removing misc-snap 2017-11-22 20:00:04,802 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 22 Nov '17

22 Nov '17

2017-11-21 20:00:02,259 INFO force is enabled 2017-11-21 20:00:02,279 INFO removing tools-project-backup 2017-11-21 20:00:02,314 INFO removing tools-project-backup 2017-11-21 20:00:02,761 INFO creating tools-project-backup at 2T 2017-11-21 20:00:03,508 INFO force is enabled 2017-11-21 20:00:03,552 INFO removing tools-snap 2017-11-21 20:00:03,598 INFO removing tools-snap 2017-11-21 20:00:05,243 INFO creating tools-snap at 1T

1 0

Cloud Clinic Notes 14/11/2017 - 21/22/2017
by Chase Pettet 20 Nov '17

20 Nov '17

1 0

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 15 Nov '17

15 Nov '17

2017-11-15 20:00:02,267 INFO force is enabled 2017-11-15 20:00:02,300 INFO removing misc-project-backup 2017-11-15 20:00:02,391 INFO removing misc-project-backup 2017-11-15 20:00:02,781 INFO creating misc-project-backup at 2T 2017-11-15 20:00:03,627 INFO force is enabled 2017-11-15 20:00:03,664 INFO removing misc-snap 2017-11-15 20:00:03,736 INFO removing misc-snap 2017-11-15 20:00:04,012 INFO creating misc-snap at 1T

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-admin November 2017