- Cloud-admin - lists.wikimedia.org

Labservices1001 just died
by Andrew Bogott 03 Jun '18

03 Jun '18

I just rebooted labservices1001 (our primary DNS server, among other things) from the console after it stopped responding to ssh. As far as I know, ordinary cloud services were unaffected by this since resolutions should have fallen back on 1002 gracefully. Puppet was upset because it explicitly names the resolver as labservices1001. I suspect this is a bad fan or heat sink installation -- the failure is https://phabricator.wikimedia.org/T196252. As best I can tell, the issue is resolved for now. You can expect a flood of recovery emails from shinken over the next 30 minutes or so. My real concern, though, is that we didn't get paged when this box locked up. I remain baffled by what pages and what doesn't; does anyone out there know how I can turn on paging for a host and/or subscribe to alerts? -A

1 0

labtestn sanity checks, weird errors
by Arturo Borrero Gonzalez 01 Jun '18

01 Jun '18

Hi, I've been working on some sanity checks for our labtestn (mitaka/neutron) deployment [0]. I found a weird issue when trying to create an instance in the 'tooling' network (the vxlan based network): << Network requires port_security_enabled and subnet associated in order to apply security groups. >> This error is seen in the /var/log/nova/nova-conductor.log file when I create an instance with this cmdline: % openstack server create --flavor 2 \ --image 66e544e8-fe4f-41f7-9809-6723e53b5a99 aborrero-test1 \ --nic port-id=9389a984-d58e-4776-8b7a-30ff93073917 \ --property subnet=3ec06de7-3b9e-4de3-86c6-67ba1895b253 However: % neutron port-show 9389a984-d58e-4776-8b7a-30ff93073917 \ | grep port_security_enabled | port_security_enabled | True (this port was created manually by me) Not sure if the 'server create' command is lacking some additional option. I generated it following what I saw in our bootstrap docs + labtestcontrol2003 history I also tried with this command: % openstack server create --flavor 2 \ --image 66e544e8-fe4f-41f7-9809-6723e53b5a99 aborrero-test1 \ --nic net-id=60aa9467-253c-4fdf-9fa0-eba42dafc975 with the same result (i.e, a net instead of a port) I'm probably misunderstanding some openstack concepts: nics, ports, subnets, nets, etc. [0] https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Deployment_sanit…

2 3

Wikitech sidebar massacre
by Andrew Bogott 31 May '18

31 May '18

I'm about to change the wikitech sidebar to remove all OSM page links. This is meant as a trial separation -- let's see if we can get along without things before actually ripping out the code. If anyone has sudden, acute regrets about this change, it can be reverted via the wiki history here: https://wikitech.wikimedia.org/wiki/MediaWiki:Sidebar/Group:projectadmin And, the actual special pages (e.g. https://wikitech.wikimedia.org/wiki/Special:NovaProject) will remain for a bit. -Andrew

1 0

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 31 May '18

31 May '18

2018-05-30 20:00:02,667 INFO force is enabled 2018-05-30 20:00:02,736 INFO removing misc-project-backup 2018-05-30 20:00:02,853 INFO removing misc-project-backup 2018-05-30 20:00:03,327 INFO creating misc-project-backup at 2T 2018-05-30 20:00:04,246 INFO force is enabled 2018-05-30 20:00:04,285 INFO removing misc-snap 2018-05-30 20:00:04,319 INFO removing misc-snap 2018-05-30 20:00:04,631 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 30 May '18

30 May '18

2018-05-29 20:00:03,170 INFO force is enabled 2018-05-29 20:00:03,218 INFO removing tools-project-backup 2018-05-29 20:00:03,318 INFO removing tools-project-backup 2018-05-29 20:00:03,790 INFO creating tools-project-backup at 2T 2018-05-29 20:00:04,572 INFO force is enabled 2018-05-29 20:00:04,598 INFO removing tools-snap 2018-05-29 20:00:04,643 INFO removing tools-snap 2018-05-29 20:00:05,799 INFO creating tools-snap at 1T

1 0

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 24 May '18

24 May '18

2018-05-23 20:00:03,063 INFO force is enabled 2018-05-23 20:00:03,129 INFO removing misc-project-backup 2018-05-23 20:00:03,229 INFO removing misc-project-backup 2018-05-23 20:00:03,687 INFO creating misc-project-backup at 2T 2018-05-23 20:00:04,773 INFO force is enabled 2018-05-23 20:00:04,880 INFO removing misc-snap 2018-05-23 20:00:04,953 INFO removing misc-snap 2018-05-23 20:00:05,326 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 23 May '18

23 May '18

2018-05-22 20:00:02,640 INFO force is enabled 2018-05-22 20:00:02,691 INFO removing tools-project-backup 2018-05-22 20:00:02,791 INFO removing tools-project-backup 2018-05-22 20:00:03,364 INFO creating tools-project-backup at 2T 2018-05-22 20:00:04,146 INFO force is enabled 2018-05-22 20:00:04,181 INFO removing tools-snap 2018-05-22 20:00:04,214 INFO removing tools-snap 2018-05-22 20:00:05,290 INFO creating tools-snap at 1T

1 0

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 17 May '18

17 May '18

2018-05-16 20:00:02,707 INFO force is enabled 2018-05-16 20:00:02,738 INFO removing misc-project-backup 2018-05-16 20:00:02,813 INFO removing misc-project-backup 2018-05-16 20:00:03,255 INFO creating misc-project-backup at 2T 2018-05-16 20:00:04,191 INFO force is enabled 2018-05-16 20:00:04,217 INFO removing misc-snap 2018-05-16 20:00:04,253 INFO removing misc-snap 2018-05-16 20:00:04,548 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 16 May '18

16 May '18

2018-05-15 20:00:02,774 INFO force is enabled 2018-05-15 20:00:02,840 INFO removing tools-project-backup 2018-05-15 20:00:02,940 INFO removing tools-project-backup 2018-05-15 20:00:03,545 INFO creating tools-project-backup at 2T 2018-05-15 20:00:04,370 INFO force is enabled 2018-05-15 20:00:04,403 INFO removing tools-snap 2018-05-15 20:00:04,462 INFO removing tools-snap 2018-05-15 20:00:05,916 INFO creating tools-snap at 1T

1 0

On call notes 2018-05-15
by Andrew Bogott 15 May '18

15 May '18

Support things: - I approved a bunch of new Tools accounts. My criteria for approval here is pretty much "If they something that sounds wikimedia-related I approve" -- if anyone else would prefer a higher bar please let me know. - I contacted people in the 'services' project and with their permission deleted 12 out of 13 VMs in that project. \o/ - Deleted the 'massmessage' project - Responded to report of a stuck job on tools-exec-1414; that host was totally stuck and unreachable. I did a hard reboot and it came back but there was nothing at all revealing in the syslog. - There were more than the usual number of "I'm confused about my ldap account" questions this week. I don't know exactly what docs they're using but I'm concerned that the renaming from 'wikitech login' to 'developer account' is causing confusion someplace. - Volunteer Annika aka Giftpflanze of giftbot jumped in and provided useful IRC support on more than one occasion this week while I was sleeping. - Chico is still helping out some too. SRE meeting things: - Faidon will look at our IP range allocation for Neutron sometime soon. - The next SRE meeting is cancelled due to European holidays. The one after that might also be cancelled as well due to US Memorial Day, but that's yet undecided.

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-admin