Cloud-admin February 2018

cloud-admin@lists.wikimedia.org

9 participants
17 discussions

clinic duty week report
by Arturo Borrero 13 Feb '18

13 Feb '18

Hi team, this is my second time doing clinic duty. Please, comment anything I may need to improve. The report follows. Timeframe: from 2018-02-06 to 2018-02-13 SRE meeting 2018-02-12: * one hour earlier this week * it was commented that we are at maximum (~25) google hangout capacity for the meeting * welcome new SRE, Valentin from Spain (Barcelona) traffic team * FY18-19 capex warning (I didn't understand this part, sorry) * no meeting next week due to US bank holiday * no questions or issues for WMCS * I mentioned we probably need some help by RobH regarding a NIC (RE: https://phabricator.wikimedia.org/T183937#3947621) * notes: https://office.wikimedia.org/wiki/Operations/Operations_Meeting_Notes/SRE-2… Clinic duty notes: * Lots of IRC helps by Chico, thanks! * Several toolforge access request approved, like 6 or 7. ** Perhaps I should have requested feedback in #236? not sure * Several bug opened, triagged, ping'ed, reviewed or done T186689 T187042 T186685 T186230 T185314 T187189 * Discussion in #wikimedia-cloud with Bryan and addshore about adding group as project admins * The usual shinken/cronspam, with no pages or important issues * I reviewed IRC channels several times during weekend. They were mostly quiet.

1 0

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 08 Feb '18

08 Feb '18

2018-02-07 20:00:03,226 INFO force is enabled 2018-02-07 20:00:03,279 INFO removing misc-project-backup 2018-02-07 20:00:03,396 INFO removing misc-project-backup 2018-02-07 20:00:04,974 INFO creating misc-project-backup at 2T 2018-02-07 20:00:05,823 INFO force is enabled 2018-02-07 20:00:05,841 INFO removing misc-snap 2018-02-07 20:00:05,913 INFO removing misc-snap 2018-02-07 20:00:06,246 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 07 Feb '18

07 Feb '18

2018-02-06 20:00:02,553 INFO force is enabled 2018-02-06 20:00:02,581 INFO removing tools-project-backup 2018-02-06 20:00:02,623 INFO removing tools-project-backup 2018-02-06 20:00:03,006 INFO creating tools-project-backup at 2T 2018-02-06 20:00:03,716 INFO force is enabled 2018-02-06 20:00:03,733 INFO removing tools-snap 2018-02-06 20:00:03,767 INFO removing tools-snap 2018-02-06 20:00:05,255 INFO creating tools-snap at 1T

1 0

Ops meeting and Clinic duty update 2/6
by Madhumitha Viswanathan 07 Feb '18

07 Feb '18

Hey all, Thanks to Chico, I did hardly any irc responding - I'm hoping he can send out his update as a follow up. Grateful for your awesomeness, Chico! *Ops meeting notes:* - Annual Planning: - Capex for next year - rough draft due this week, discussion in tech mgmt meeting next week - Strengthening infrastructure is a focus for next annual plan - Fosdem last weekend - Riccardo and Jaime presented - TLS for MySQL at large scale: https://fosdem.org/2018/schedule/event/tls_mysql/ - Cumin: https://fosdem.org/2018/schedule/event/cumin_automation/ - Next week SRE meeting is at a slightly different time due to conflicts - Gerrit UI upgrade upcoming - check it out with ?polygerrit=1 on the url (and polygerrit=0 to disable) *Clinic duty notes:* - Icinga access for Brooke and missed access for Arturo - https://gerrit.wikimedia.org/r/#/c/408340/ - https://gerrit.wikimedia.org/r/40832 <https://gerrit.wikimedia.org/r/408328> - https://gerrit.wikimedia.org/r/408327 - labsdb1001/3 decommission - merged toolschecker and prometheus removal patches put up by Bryan - following up on https://phabricator.wikimedia.org/T183029 - Quota increase for https://phabricator.wikimedia.org/T185851 - PAWS user home hack - submitted script to grid, but it seems to throw a ton of errors, poked Yuvi on it (pending investigation) - Fixed failing tools and misc dumps - https://phabricator.wikimedia.org/T186235 - Granted some toolforge access requests Best, -- Madhumitha Viswanathan Operations Engineer, Cloud Services

2 1

Outreachy application window opening soon
by Bryan Davis 04 Feb '18

04 Feb '18

Are we interested in trying an Outreachy project again? The next round opens February 12, so we need to decide soon if we want to get in. Bryan -- Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org> [[m:User:BDavis_(WMF)]] Manager, Cloud Services Boise, ID USA irc: bd808 v:415.839.6885 x6855

1 0

Fwd: Cron <root@tools-exec-1436> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
by Chase Pettet 01 Feb '18

01 Feb '18

Anyone know why this keep happening on this host? Arturo? :) ---------- Forwarded message ---------- From: Cron Daemon <root(a)tools.wmflabs.org> Date: Wed, Jan 31, 2018 at 12:47 AM Subject: Cron <root@tools-exec-1436> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) To: root(a)tools.wmflabs.org /etc/cron.daily/apt-show-versions: Can't open file /var/lib/apt/lists/nova.clouds.archive.ubuntu.com_ ubuntu_dists_trusty_universe_binary-amd64_Packages: No such file or directory run-parts: /etc/cron.daily/apt-show-versions exited with return code 255 -- Chase Pettet chasemp on phabricator <https://phabricator.wikimedia.org/p/chasemp/> and IRC

2 2

Fwd: ** RECOVERY alert - tools-webgrid-lighttpd-1416/High iowait is OK **
by Chase Pettet 01 Feb '18

01 Feb '18

https://phabricator.wikimedia.org/T186222 ---------- Forwarded message ---------- From: shinken <shinken(a)shinken-01.shinken.eqiad.wmflabs> Date: Thu, Feb 1, 2018 at 8:40 AM Subject: ** RECOVERY alert - tools-webgrid-lighttpd-1416/High iowait is OK ** To: cpettet(a)wikimedia.org Notification Type: RECOVERY Service: High iowait Host: tools-webgrid-lighttpd-1416 Address: 10.68.19.50 State: OK Date/Time: Thu 01 Feb 14:40:38 UTC 2018 Notes URLs: Additional Info: OK: All targets OK -- Chase Pettet chasemp on phabricator <https://phabricator.wikimedia.org/p/chasemp/> and IRC

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-admin February 2018