June 2018 - Cloud-admin - lists.wikimedia.org

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 27 Jun '18

27 Jun '18

2018-06-27 20:00:03,212 INFO force is enabled 2018-06-27 20:00:03,262 INFO removing misc-project-backup 2018-06-27 20:00:03,371 INFO removing misc-project-backup 2018-06-27 20:00:03,819 INFO creating misc-project-backup at 2T 2018-06-27 20:00:04,751 INFO force is enabled 2018-06-27 20:00:04,801 INFO removing misc-snap 2018-06-27 20:00:04,844 INFO removing misc-snap 2018-06-27 20:00:05,148 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 27 Jun '18

27 Jun '18

2018-06-26 20:00:03,103 INFO force is enabled 2018-06-26 20:00:03,155 INFO removing tools-project-backup 2018-06-26 20:00:03,254 INFO removing tools-project-backup 2018-06-26 20:00:03,810 INFO creating tools-project-backup at 2T 2018-06-26 20:00:04,653 INFO force is enabled 2018-06-26 20:00:04,693 INFO removing tools-snap 2018-06-26 20:00:04,755 INFO removing tools-snap 2018-06-26 20:00:06,585 INFO creating tools-snap at 1T

1 0

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 20 Jun '18

20 Jun '18

2018-06-20 20:00:02,777 INFO force is enabled 2018-06-20 20:00:02,835 INFO removing misc-project-backup 2018-06-20 20:00:02,951 INFO removing misc-project-backup 2018-06-20 20:00:03,423 INFO creating misc-project-backup at 2T 2018-06-20 20:00:04,303 INFO force is enabled 2018-06-20 20:00:04,326 INFO removing misc-snap 2018-06-20 20:00:04,377 INFO removing misc-snap 2018-06-20 20:00:04,696 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 20 Jun '18

20 Jun '18

2018-06-19 20:00:02,282 INFO force is enabled 2018-06-19 20:00:02,328 INFO removing tools-project-backup 2018-06-19 20:00:02,427 INFO removing tools-project-backup 2018-06-19 20:00:03,008 INFO creating tools-project-backup at 2T 2018-06-19 20:00:03,885 INFO force is enabled 2018-06-19 20:00:03,916 INFO removing tools-snap 2018-06-19 20:00:03,961 INFO removing tools-snap 2018-06-19 20:00:05,591 INFO creating tools-snap at 1T

1 0

Fwd: Deployment-prep Puppet error hosts report
by Alex Monk 15 Jun '18

15 Jun '18

Thought I'd mention something I worked on recently - I have a cron on deployment-cumin that runs puppet across everything (that openstack says is running) and emails me with a list of hosts with any problems. It has a little config allowing associating a host with a task. deployment-prep is looking better than I thought. (deploy-01 got broken in the security updates and I'm planning to look into cache-text04 later - think this was a timeout of some sort, likely related to my certificate work there) ---------- Forwarded message ---------- From: <krenair(a)beta.wmflabs.org> Date: 15 June 2018 at 19:05 Subject: Deployment-prep Puppet error hosts report To: krenair(a)gmail.com HostnameTask? deployment-cache-text04.deployment-prep.eqiad.wmflabs None deployment-deploy-01.deployment-prep.eqiad.wmflabs T192561 <https://phabricator.wikimedia.org/T192561> Hosts configured with tasks but are not listing as broken anymore: HostnameTask deployment-mx.deployment-prep.eqiad.wmflabs T184244

2 2

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 13 Jun '18

13 Jun '18

2018-06-13 20:00:02,511 INFO force is enabled 2018-06-13 20:00:02,577 INFO removing misc-project-backup 2018-06-13 20:00:02,694 INFO removing misc-project-backup 2018-06-13 20:00:03,342 INFO creating misc-project-backup at 2T 2018-06-13 20:00:04,355 INFO force is enabled 2018-06-13 20:00:04,480 INFO removing misc-snap 2018-06-13 20:00:04,527 INFO removing misc-snap 2018-06-13 20:00:04,923 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 13 Jun '18

13 Jun '18

2018-06-12 20:00:02,989 INFO force is enabled 2018-06-12 20:00:03,036 INFO removing tools-project-backup 2018-06-12 20:00:03,135 INFO removing tools-project-backup 2018-06-12 20:00:03,775 INFO creating tools-project-backup at 2T 2018-06-12 20:00:04,578 INFO force is enabled 2018-06-12 20:00:04,599 INFO removing tools-snap 2018-06-12 20:00:04,657 INFO removing tools-snap 2018-06-12 20:00:06,107 INFO creating tools-snap at 1T

1 0

SRE meeting 11-06-2018
by Chase Pettet 11 Jun '18

11 Jun '18

* Andrew B could you look at https://phabricator.wikimedia.org/T188377 luka is worried that cleaning up submodules will mess with puppetmasters in cloud * Cloud Services is asked to do a 10m presentation on state of things and forward thinking in Prague. I volunteered to own this so I'll try to share something out. Nothing too fancy, highlights to prompt discussion seems appropriate. * I mentioned that none of the 10G ports on the HP's we have bought recently actually seem to work. This includes labvirt1019, labvirt1020, labnet1003, labnet1004 IIUC. There have been different tasks related over time but we had not put together the common thread until this morning. I'll make a new distinct task and try to line things up for soliciting wider help. * We really need to settle on some dates for moving lab* servers to new switches, they are the last remaining in row b. At this point it seems likely to be post-prague though. * Don't forget to update https://etherpad.wikimedia.org/p/WMCS-2018-06-12 * Weird week with lots of travel all around -- Chase Pettet chasemp on phabricator <https://phabricator.wikimedia.org/p/chasemp/> and IRC

1 0

cloud team norms update(s)
by Chase Pettet 07 Jun '18

07 Jun '18

There has been increased interest in the team social norms that were codified last year. Deb Tankersley even gave a presentation to all tech managers referencing our work here (Thanks Deb!). I have had a few things on my bucket list in regards to these as far as editing, and adding context I found myself consistently giving in casual conversation. I'm hoping the team can reread these as they stand now and we can iterate on any inconsistencies I have unearthed :). It has been the case thus far that I do the majority of the writing and ...some would say overt pontification :D but I'm hoping I have not strayed from our shared original purpose. https://www.mediawiki.org/wiki/Wikimedia_Cloud_Services_team/Team_Social_No… https://www.mediawiki.org/w/index.php?title=Wikimedia_Cloud_Services_team%2… I feel better about socializing this beyond our own borders now, and thanks to all of you for giving this credence to this point. I think the 'no belittling' section could use a little polish but I hope to keep it around in general. Best, -- Chase Pettet chasemp on phabricator <https://phabricator.wikimedia.org/p/chasemp/> and IRC

2 3

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 06 Jun '18

06 Jun '18

2018-06-06 20:00:02,180 INFO force is enabled 2018-06-06 20:00:02,243 INFO removing misc-project-backup 2018-06-06 20:00:02,360 INFO removing misc-project-backup 2018-06-06 20:00:02,850 INFO creating misc-project-backup at 2T 2018-06-06 20:00:03,783 INFO force is enabled 2018-06-06 20:00:03,829 INFO removing misc-snap 2018-06-06 20:00:03,936 INFO removing misc-snap 2018-06-06 20:00:04,417 INFO creating misc-snap at 1T

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-admin June 2018