- Cloud-admin - lists.wikimedia.org

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 26 Oct '17

26 Oct '17

2017-10-25 20:00:02,697 INFO force is enabled 2017-10-25 20:00:02,741 INFO removing misc-project-backup 2017-10-25 20:00:02,815 INFO removing misc-project-backup 2017-10-25 20:00:03,206 INFO creating misc-project-backup at 2T 2017-10-25 20:00:04,087 INFO force is enabled 2017-10-25 20:00:04,141 INFO removing misc-snap 2017-10-25 20:00:04,183 INFO removing misc-snap 2017-10-25 20:00:04,470 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 25 Oct '17

25 Oct '17

2017-10-24 20:00:03,279 INFO force is enabled 2017-10-24 20:00:03,317 INFO removing tools-project-backup 2017-10-24 20:00:03,400 INFO removing tools-project-backup 2017-10-24 20:00:03,998 INFO creating tools-project-backup at 2T 2017-10-24 20:00:04,831 INFO force is enabled 2017-10-24 20:00:04,867 INFO removing tools-snap 2017-10-24 20:00:04,902 INFO removing tools-snap 2017-10-24 20:00:06,293 INFO creating tools-snap at 1T

1 0

Clinic Oct 17 - 24
by Chase Pettet 25 Oct '17

25 Oct '17

Ops Meeting * Sometime in the next few months Mediawiki and Trusty are no longer friends. I reiterated Andrew is aware of the precarious position of silver and wikitech and it's on the books to handle. * Mark is thinking of possibly a mini-ops-offsite for a day or two post all hands. Nothing seems set in stone yet. * Arturo is waiting on confirmation from Legal for one last thing then I'll merge https://gerrit.wikimedia.org/r/#/c/385988/ as it was approved in this weeks meeting. Misc: * Added a new thing to start archiving meeting notes https://office.wikimedia.org/wiki/Wikimedia_Cloud_Services/Meeting_Notes/20… :* Meeting template https://office.wikimedia.org/wiki/Wikimedia_Cloud_Services/Meeting_Notes/te… Weekly: * Finally closed https://phabricator.wikimedia.org/T169774 * Bumped up Tools quota https://phabricator.wikimedia.org/T178805 * Cyberbot quota increase(s) https://phabricator.wikimedia.org/T178332 * New Tools elastic creds process was fine https://phabricator.wikimedia.org/T178310 * misc support in chat, per usual Bryan did a lot of it :) Outstanding: * https://phabricator.wikimedia.org/T178837 :* I think https://phabricator.wikimedia.org/T178837#3708165 means it was a one-time script and so debugging it is going to be pretty difficult or pointless? * https://phabricator.wikimedia.org/T178803 :* I'm not clear on what is happening w/ wsexport or if it's still happening or exactly how to debug. I spent some time on it when it was reported but I guess epub does not use the grid so my testing wasn't useful (https://phabricator.wikimedia.org/T178803#3703633) and I didn't get time to come back to this. * Labvirt1015 still stinks. Let's hope a new board fixes things. https://phabricator.wikimedia.org/T171473 * https://phabricator.wikimedia.org/T177944 :* I think there is consensus that rebase as a mode of keeping cherry-picks but following upstream has this race condition and any secret sitting on top of our branch will have this issue. I'm not sure if we have a viable alternative but at least we believe we understand the issue. -- Chase Pettet chasemp on phabricator <https://phabricator.wikimedia.org/p/chasemp/> and IRC

1 0

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 19 Oct '17

19 Oct '17

2017-10-18 20:00:03,404 INFO force is enabled 2017-10-18 20:00:03,438 INFO removing misc-project-backup 2017-10-18 20:00:03,513 INFO removing misc-project-backup 2017-10-18 20:00:04,019 INFO creating misc-project-backup at 2T 2017-10-18 20:00:04,931 INFO force is enabled 2017-10-18 20:00:04,956 INFO removing misc-snap 2017-10-18 20:00:04,989 INFO removing misc-snap 2017-10-18 20:00:05,355 INFO creating misc-snap at 1T

1 0

small reminder: I'm away for the next 4 days
by Chase Pettet 19 Oct '17

19 Oct '17

I'll be driving or doing things around St. Louis. I'll keep my gear in the trunk but my status will be in flux. -- Chase Pettet chasemp on phabricator <https://phabricator.wikimedia.org/p/chasemp/> and IRC

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 18 Oct '17

18 Oct '17

2017-10-17 20:00:02,943 INFO force is enabled 2017-10-17 20:00:02,981 INFO removing tools-project-backup 2017-10-17 20:00:03,090 INFO removing tools-project-backup 2017-10-17 20:00:03,688 INFO creating tools-project-backup at 2T 2017-10-17 20:00:04,487 INFO force is enabled 2017-10-17 20:00:04,507 INFO removing tools-snap 2017-10-17 20:00:04,555 INFO removing tools-snap 2017-10-17 20:00:05,638 INFO creating tools-snap at 1T

1 0

on-call notes: 2017-10-10 - 2017-10-16
by Andrew Bogott 18 Oct '17

18 Oct '17

No Ops meeting this week. * T176043: Prepare and check storage layer for amwikimedia ** Added amwikimedia to replicas * T70876: Make user_email_authenticated status visible on labs ** Fixed (bd808 did most of the work, I just merged and updated dbs) * T173891: Create core ip_changes view for replicas ** Again, just merged and applied bd808's changes * Created mwstake project * Deleted OCG project * Temporarily raised IP quota for CVN * Not much support this week. Bryan chatted with Dispenser quite a bit and I stayed out of it.

2 1

on-call notes: 2017-10-03 - 2017-10-09
by Bryan Davis 13 Oct '17

13 Oct '17

== 2017-10-03 - 2017-10-09 == Tues: * Created iiab project * PAWS returning 503s on login ** Turned out to be iptables problems of mysterious origin Wed: * cloud-admin@ non-member filter adjustments * fixed some little issues in Phab project descriptions Thur: * talked with Freddy2001 and others about Freddy2001's Varnish 503 issues * setup freenode config for new #wikimedia-cloud-feed channel * helped Andrew figure out why wikitech-static was missing content * Deployed a Striker patch to disallow tool names that k8s can't use as namespace labels * build and deployed new jobsutils and misctools packages Fri: * Saw report of jstart not working. Fixed the problem I had caused. * new project request triage * rubber ducked while raynor/Pmiazga figured out how to ssh to an instance Weekend: * nice a quiet Mon: * Foundation holiday, also pretty quiet There was no ops meeting due to the holiday. +1 again for our move to Tuesdays. Bryan -- Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org> [[m:User:BDavis_(WMF)]] Manager, Cloud Services Boise, ID USA irc: bd808 v:415.839.6885 x6855

1 0

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 12 Oct '17

12 Oct '17

2017-10-11 20:00:02,790 INFO force is enabled 2017-10-11 20:00:02,822 INFO removing misc-project-backup 2017-10-11 20:00:02,891 INFO removing misc-project-backup 2017-10-11 20:00:03,414 INFO creating misc-project-backup at 2T 2017-10-11 20:00:04,370 INFO force is enabled 2017-10-11 20:00:04,411 INFO removing misc-snap 2017-10-11 20:00:04,465 INFO removing misc-snap 2017-10-11 20:00:04,852 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 11 Oct '17

11 Oct '17

2017-10-10 20:00:03,208 INFO force is enabled 2017-10-10 20:00:03,254 INFO removing tools-project-backup 2017-10-10 20:00:03,362 INFO removing tools-project-backup 2017-10-10 20:00:03,961 INFO creating tools-project-backup at 2T 2017-10-10 20:00:04,787 INFO force is enabled 2017-10-10 20:00:04,813 INFO removing tools-snap 2017-10-10 20:00:04,861 INFO removing tools-snap 2017-10-10 20:00:05,933 INFO creating tools-snap at 1T

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-admin