May 2018 - Cloud-admin - lists.wikimedia.org

Wikitech sidebar massacre
by Andrew Bogott 31 May '18

31 May '18

I'm about to change the wikitech sidebar to remove all OSM page links. This is meant as a trial separation -- let's see if we can get along without things before actually ripping out the code. If anyone has sudden, acute regrets about this change, it can be reverted via the wiki history here: https://wikitech.wikimedia.org/wiki/MediaWiki:Sidebar/Group:projectadmin And, the actual special pages (e.g. https://wikitech.wikimedia.org/wiki/Special:NovaProject) will remain for a bit. -Andrew

1 0

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 31 May '18

31 May '18

2018-05-30 20:00:02,667 INFO force is enabled 2018-05-30 20:00:02,736 INFO removing misc-project-backup 2018-05-30 20:00:02,853 INFO removing misc-project-backup 2018-05-30 20:00:03,327 INFO creating misc-project-backup at 2T 2018-05-30 20:00:04,246 INFO force is enabled 2018-05-30 20:00:04,285 INFO removing misc-snap 2018-05-30 20:00:04,319 INFO removing misc-snap 2018-05-30 20:00:04,631 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 30 May '18

30 May '18

2018-05-29 20:00:03,170 INFO force is enabled 2018-05-29 20:00:03,218 INFO removing tools-project-backup 2018-05-29 20:00:03,318 INFO removing tools-project-backup 2018-05-29 20:00:03,790 INFO creating tools-project-backup at 2T 2018-05-29 20:00:04,572 INFO force is enabled 2018-05-29 20:00:04,598 INFO removing tools-snap 2018-05-29 20:00:04,643 INFO removing tools-snap 2018-05-29 20:00:05,799 INFO creating tools-snap at 1T

1 0

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 24 May '18

24 May '18

2018-05-23 20:00:03,063 INFO force is enabled 2018-05-23 20:00:03,129 INFO removing misc-project-backup 2018-05-23 20:00:03,229 INFO removing misc-project-backup 2018-05-23 20:00:03,687 INFO creating misc-project-backup at 2T 2018-05-23 20:00:04,773 INFO force is enabled 2018-05-23 20:00:04,880 INFO removing misc-snap 2018-05-23 20:00:04,953 INFO removing misc-snap 2018-05-23 20:00:05,326 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 23 May '18

23 May '18

2018-05-22 20:00:02,640 INFO force is enabled 2018-05-22 20:00:02,691 INFO removing tools-project-backup 2018-05-22 20:00:02,791 INFO removing tools-project-backup 2018-05-22 20:00:03,364 INFO creating tools-project-backup at 2T 2018-05-22 20:00:04,146 INFO force is enabled 2018-05-22 20:00:04,181 INFO removing tools-snap 2018-05-22 20:00:04,214 INFO removing tools-snap 2018-05-22 20:00:05,290 INFO creating tools-snap at 1T

1 0

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 17 May '18

17 May '18

2018-05-16 20:00:02,707 INFO force is enabled 2018-05-16 20:00:02,738 INFO removing misc-project-backup 2018-05-16 20:00:02,813 INFO removing misc-project-backup 2018-05-16 20:00:03,255 INFO creating misc-project-backup at 2T 2018-05-16 20:00:04,191 INFO force is enabled 2018-05-16 20:00:04,217 INFO removing misc-snap 2018-05-16 20:00:04,253 INFO removing misc-snap 2018-05-16 20:00:04,548 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 16 May '18

16 May '18

2018-05-15 20:00:02,774 INFO force is enabled 2018-05-15 20:00:02,840 INFO removing tools-project-backup 2018-05-15 20:00:02,940 INFO removing tools-project-backup 2018-05-15 20:00:03,545 INFO creating tools-project-backup at 2T 2018-05-15 20:00:04,370 INFO force is enabled 2018-05-15 20:00:04,403 INFO removing tools-snap 2018-05-15 20:00:04,462 INFO removing tools-snap 2018-05-15 20:00:05,916 INFO creating tools-snap at 1T

1 0

On call notes 2018-05-15
by Andrew Bogott 15 May '18

15 May '18

Support things: - I approved a bunch of new Tools accounts. My criteria for approval here is pretty much "If they something that sounds wikimedia-related I approve" -- if anyone else would prefer a higher bar please let me know. - I contacted people in the 'services' project and with their permission deleted 12 out of 13 VMs in that project. \o/ - Deleted the 'massmessage' project - Responded to report of a stuck job on tools-exec-1414; that host was totally stuck and unreachable. I did a hard reboot and it came back but there was nothing at all revealing in the syslog. - There were more than the usual number of "I'm confused about my ldap account" questions this week. I don't know exactly what docs they're using but I'm concerned that the renaming from 'wikitech login' to 'developer account' is causing confusion someplace. - Volunteer Annika aka Giftpflanze of giftbot jumped in and provided useful IRC support on more than one occasion this week while I was sleeping. - Chico is still helping out some too. SRE meeting things: - Faidon will look at our IP range allocation for Neutron sometime soon. - The next SRE meeting is cancelled due to European holidays. The one after that might also be cancelled as well due to US Memorial Day, but that's yet undecided.

1 0

wikireplicas reaching maximum utilization
by Jaime Crespo 10 May '18

10 May '18

https://grafana.wikimedia.org/dashboard/file/server-board.json?refresh=1m&p… What to do? -- Jaime Crespo <http://wikimedia.org>

2 2

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 10 May '18

10 May '18

2018-05-09 20:00:02,600 INFO force is enabled 2018-05-09 20:00:02,657 INFO removing misc-project-backup 2018-05-09 20:00:02,773 INFO removing misc-project-backup 2018-05-09 20:00:03,631 INFO creating misc-project-backup at 2T 2018-05-09 20:00:04,574 INFO force is enabled 2018-05-09 20:00:04,634 INFO removing misc-snap 2018-05-09 20:00:04,697 INFO removing misc-snap 2018-05-09 20:00:04,988 INFO creating misc-snap at 1T

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-admin May 2018