October 2017 - Cloud-admin - lists.wikimedia.org

Clinic duty and Ops meeting update 10/30
by Madhumitha Viswanathan 30 Oct '17

30 Oct '17

Ops meeting notes: https://office.wikimedia.org/wiki/Operations/ Operations_Meeting_Notes/TechOps-2017-10-30 *Updates*: * There is a mini ops offsite planned for the weekend after all hands, it's Tech Ops only (not us), max limit 18, and voluntary. * Initial annual planning is happening now for board to approve the budget envelope for next year. Programs and major Cap-ex stuff may be set up now, reach out to Mark if you have something to suggest/add here * Giuseppe had a question about when we are going to turn on the future parser for puppet in instances, I said I'd find out and update - if you know, may be ping him, or update a ticket :) *Clinic duty notes*: * Labsdb reboots planning - https://phabricator.wikimedia.org/T168584. Labsdb1001 is done now, 1003 is scheduled for next week * Followed up on tools-package-builder puppet failures, Alex resolved it by upgrading a package - https://phabricator.wikimedia.org/T178920 * Set up the labsdbadmin user with powers to access raw database tables, with help from Manuel. See - https://phabricator.wikimedia.org/T178128. Also note that we are tracking Wiki Replica specific grants in puppet now at https://github.com/wikimedia/puppet/blob/production/modules/role/ templates/mariadb/grants/wiki-replicas.sql * Maintain-dbusers failure due to old account user clash resurfaced, fixed - https://phabricator.wikimedia.org/T179075 * Helped releng run a command across deployment-prep using cumin * Resolved some Toolforge access requests * Helped an outreachy applicant run their tool on Toolforge over the weekend * Other IRC support and patrolling over the week(end) -- Madhumitha Viswanathan Operations Engineer, Cloud Services

1 0

cloud@ set to automatically discard non-member posts
by Bryan Davis 26 Oct '17

26 Oct '17

Non-member posts to cloud@ are basically always spam. I'm tried of seeing them, so I set the list to auto discard non-member posts and send this response to the poster: Sorry, non-member posts are not accepted by this list. See <https://lists.wikimedia.org/mailman/listinfo/cloud> for instructions on becoming a list member. Bryan -- Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org> [[m:User:BDavis_(WMF)]] Manager, Cloud Services Boise, ID USA irc: bd808 v:415.839.6885 x6855

1 0

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 25 Oct '17

25 Oct '17

2017-10-25 20:00:02,697 INFO force is enabled 2017-10-25 20:00:02,741 INFO removing misc-project-backup 2017-10-25 20:00:02,815 INFO removing misc-project-backup 2017-10-25 20:00:03,206 INFO creating misc-project-backup at 2T 2017-10-25 20:00:04,087 INFO force is enabled 2017-10-25 20:00:04,141 INFO removing misc-snap 2017-10-25 20:00:04,183 INFO removing misc-snap 2017-10-25 20:00:04,470 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 25 Oct '17

25 Oct '17

2017-10-24 20:00:03,279 INFO force is enabled 2017-10-24 20:00:03,317 INFO removing tools-project-backup 2017-10-24 20:00:03,400 INFO removing tools-project-backup 2017-10-24 20:00:03,998 INFO creating tools-project-backup at 2T 2017-10-24 20:00:04,831 INFO force is enabled 2017-10-24 20:00:04,867 INFO removing tools-snap 2017-10-24 20:00:04,902 INFO removing tools-snap 2017-10-24 20:00:06,293 INFO creating tools-snap at 1T

1 0

Clinic Oct 17 - 24
by Chase Pettet 24 Oct '17

24 Oct '17

Ops Meeting * Sometime in the next few months Mediawiki and Trusty are no longer friends. I reiterated Andrew is aware of the precarious position of silver and wikitech and it's on the books to handle. * Mark is thinking of possibly a mini-ops-offsite for a day or two post all hands. Nothing seems set in stone yet. * Arturo is waiting on confirmation from Legal for one last thing then I'll merge https://gerrit.wikimedia.org/r/#/c/385988/ as it was approved in this weeks meeting. Misc: * Added a new thing to start archiving meeting notes https://office.wikimedia.org/wiki/Wikimedia_Cloud_Services/Meeting_Notes/20… :* Meeting template https://office.wikimedia.org/wiki/Wikimedia_Cloud_Services/Meeting_Notes/te… Weekly: * Finally closed https://phabricator.wikimedia.org/T169774 * Bumped up Tools quota https://phabricator.wikimedia.org/T178805 * Cyberbot quota increase(s) https://phabricator.wikimedia.org/T178332 * New Tools elastic creds process was fine https://phabricator.wikimedia.org/T178310 * misc support in chat, per usual Bryan did a lot of it :) Outstanding: * https://phabricator.wikimedia.org/T178837 :* I think https://phabricator.wikimedia.org/T178837#3708165 means it was a one-time script and so debugging it is going to be pretty difficult or pointless? * https://phabricator.wikimedia.org/T178803 :* I'm not clear on what is happening w/ wsexport or if it's still happening or exactly how to debug. I spent some time on it when it was reported but I guess epub does not use the grid so my testing wasn't useful (https://phabricator.wikimedia.org/T178803#3703633) and I didn't get time to come back to this. * Labvirt1015 still stinks. Let's hope a new board fixes things. https://phabricator.wikimedia.org/T171473 * https://phabricator.wikimedia.org/T177944 :* I think there is consensus that rebase as a mode of keeping cherry-picks but following upstream has this race condition and any secret sitting on top of our branch will have this issue. I'm not sure if we have a viable alternative but at least we believe we understand the issue. -- Chase Pettet chasemp on phabricator <https://phabricator.wikimedia.org/p/chasemp/> and IRC

1 0

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 18 Oct '17

18 Oct '17

2017-10-18 20:00:03,404 INFO force is enabled 2017-10-18 20:00:03,438 INFO removing misc-project-backup 2017-10-18 20:00:03,513 INFO removing misc-project-backup 2017-10-18 20:00:04,019 INFO creating misc-project-backup at 2T 2017-10-18 20:00:04,931 INFO force is enabled 2017-10-18 20:00:04,956 INFO removing misc-snap 2017-10-18 20:00:04,989 INFO removing misc-snap 2017-10-18 20:00:05,355 INFO creating misc-snap at 1T

1 0

small reminder: I'm away for the next 4 days
by Chase Pettet 18 Oct '17

18 Oct '17

I'll be driving or doing things around St. Louis. I'll keep my gear in the trunk but my status will be in flux. -- Chase Pettet chasemp on phabricator <https://phabricator.wikimedia.org/p/chasemp/> and IRC

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 18 Oct '17

18 Oct '17

2017-10-17 20:00:02,943 INFO force is enabled 2017-10-17 20:00:02,981 INFO removing tools-project-backup 2017-10-17 20:00:03,090 INFO removing tools-project-backup 2017-10-17 20:00:03,688 INFO creating tools-project-backup at 2T 2017-10-17 20:00:04,487 INFO force is enabled 2017-10-17 20:00:04,507 INFO removing tools-snap 2017-10-17 20:00:04,555 INFO removing tools-snap 2017-10-17 20:00:05,638 INFO creating tools-snap at 1T

1 0

on-call notes: 2017-10-10 - 2017-10-16
by Andrew Bogott 17 Oct '17

17 Oct '17

No Ops meeting this week. * T176043: Prepare and check storage layer for amwikimedia ** Added amwikimedia to replicas * T70876: Make user_email_authenticated status visible on labs ** Fixed (bd808 did most of the work, I just merged and updated dbs) * T173891: Create core ip_changes view for replicas ** Again, just merged and applied bd808's changes * Created mwstake project * Deleted OCG project * Temporarily raised IP quota for CVN * Not much support this week. Bryan chatted with Dispenser quite a bit and I stayed out of it.

2 1

on-call notes: 2017-10-03 - 2017-10-09
by Bryan Davis 12 Oct '17

12 Oct '17

== 2017-10-03 - 2017-10-09 == Tues: * Created iiab project * PAWS returning 503s on login ** Turned out to be iptables problems of mysterious origin Wed: * cloud-admin@ non-member filter adjustments * fixed some little issues in Phab project descriptions Thur: * talked with Freddy2001 and others about Freddy2001's Varnish 503 issues * setup freenode config for new #wikimedia-cloud-feed channel * helped Andrew figure out why wikitech-static was missing content * Deployed a Striker patch to disallow tool names that k8s can't use as namespace labels * build and deployed new jobsutils and misctools packages Fri: * Saw report of jstart not working. Fixed the problem I had caused. * new project request triage * rubber ducked while raynor/Pmiazga figured out how to ssh to an instance Weekend: * nice a quiet Mon: * Foundation holiday, also pretty quiet There was no ops meeting due to the holiday. +1 again for our move to Tuesdays. Bryan -- Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org> [[m:User:BDavis_(WMF)]] Manager, Cloud Services Boise, ID USA irc: bd808 v:415.839.6885 x6855

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-admin October 2017