Ops meeting notes: https://office.wikimedia.org/wiki/Operations/
Operations_Meeting_Notes/TechOps-2017-10-30
*Updates*:
* There is a mini ops offsite planned for the weekend after all hands, it's
Tech Ops only (not us), max limit 18, and voluntary.
* Initial annual planning is happening now for board to approve the budget
envelope for next year. Programs and major Cap-ex stuff may be set up now,
reach out to Mark if you have something to suggest/add here
* Giuseppe had a question about when we are going to turn on the future
parser for puppet in instances, I said I'd find out and update - if you
know, may be ping him, or update a ticket :)
*Clinic duty notes*:
* Labsdb reboots planning - https://phabricator.wikimedia.org/T168584.
Labsdb1001 is done now, 1003 is scheduled for next week
* Followed up on tools-package-builder puppet failures, Alex resolved it by
upgrading a package - https://phabricator.wikimedia.org/T178920
* Set up the labsdbadmin user with powers to access raw database tables,
with help from Manuel. See - https://phabricator.wikimedia.org/T178128.
Also note that we are tracking Wiki Replica specific grants in puppet now
at https://github.com/wikimedia/puppet/blob/production/modules/role/
templates/mariadb/grants/wiki-replicas.sql
* Maintain-dbusers failure due to old account user clash resurfaced, fixed
- https://phabricator.wikimedia.org/T179075
* Helped releng run a command across deployment-prep using cumin
* Resolved some Toolforge access requests
* Helped an outreachy applicant run their tool on Toolforge over the weekend
* Other IRC support and patrolling over the week(end)
--
Madhumitha Viswanathan
Operations Engineer, Cloud Services
Non-member posts to cloud@ are basically always spam. I'm tried of
seeing them, so I set the list to auto discard non-member posts and
send this response to the poster:
Sorry, non-member posts are not accepted by this list. See
<https://lists.wikimedia.org/mailman/listinfo/cloud> for instructions
on becoming a list member.
Bryan
--
Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org>
[[m:User:BDavis_(WMF)]] Manager, Cloud Services Boise, ID USA
irc: bd808 v:415.839.6885 x6855
2017-10-25 20:00:02,697 INFO force is enabled
2017-10-25 20:00:02,741 INFO removing misc-project-backup
2017-10-25 20:00:02,815 INFO removing misc-project-backup
2017-10-25 20:00:03,206 INFO creating misc-project-backup at 2T
2017-10-25 20:00:04,087 INFO force is enabled
2017-10-25 20:00:04,141 INFO removing misc-snap
2017-10-25 20:00:04,183 INFO removing misc-snap
2017-10-25 20:00:04,470 INFO creating misc-snap at 1T
2017-10-24 20:00:03,279 INFO force is enabled
2017-10-24 20:00:03,317 INFO removing tools-project-backup
2017-10-24 20:00:03,400 INFO removing tools-project-backup
2017-10-24 20:00:03,998 INFO creating tools-project-backup at 2T
2017-10-24 20:00:04,831 INFO force is enabled
2017-10-24 20:00:04,867 INFO removing tools-snap
2017-10-24 20:00:04,902 INFO removing tools-snap
2017-10-24 20:00:06,293 INFO creating tools-snap at 1T
2017-10-18 20:00:03,404 INFO force is enabled
2017-10-18 20:00:03,438 INFO removing misc-project-backup
2017-10-18 20:00:03,513 INFO removing misc-project-backup
2017-10-18 20:00:04,019 INFO creating misc-project-backup at 2T
2017-10-18 20:00:04,931 INFO force is enabled
2017-10-18 20:00:04,956 INFO removing misc-snap
2017-10-18 20:00:04,989 INFO removing misc-snap
2017-10-18 20:00:05,355 INFO creating misc-snap at 1T
I'll be driving or doing things around St. Louis. I'll keep my gear in the
trunk but my status will be in flux.
--
Chase Pettet
chasemp on phabricator <https://phabricator.wikimedia.org/p/chasemp/> and
IRC
2017-10-17 20:00:02,943 INFO force is enabled
2017-10-17 20:00:02,981 INFO removing tools-project-backup
2017-10-17 20:00:03,090 INFO removing tools-project-backup
2017-10-17 20:00:03,688 INFO creating tools-project-backup at 2T
2017-10-17 20:00:04,487 INFO force is enabled
2017-10-17 20:00:04,507 INFO removing tools-snap
2017-10-17 20:00:04,555 INFO removing tools-snap
2017-10-17 20:00:05,638 INFO creating tools-snap at 1T
No Ops meeting this week.
* T176043: Prepare and check storage layer for amwikimedia
** Added amwikimedia to replicas
* T70876: Make user_email_authenticated status visible on labs
** Fixed (bd808 did most of the work, I just merged and updated dbs)
* T173891: Create core ip_changes view for replicas
** Again, just merged and applied bd808's changes
* Created mwstake project
* Deleted OCG project
* Temporarily raised IP quota for CVN
* Not much support this week. Bryan chatted with Dispenser quite a bit
and I stayed out of it.
== 2017-10-03 - 2017-10-09 ==
Tues:
* Created iiab project
* PAWS returning 503s on login
** Turned out to be iptables problems of mysterious origin
Wed:
* cloud-admin@ non-member filter adjustments
* fixed some little issues in Phab project descriptions
Thur:
* talked with Freddy2001 and others about Freddy2001's Varnish 503 issues
* setup freenode config for new #wikimedia-cloud-feed channel
* helped Andrew figure out why wikitech-static was missing content
* Deployed a Striker patch to disallow tool names that k8s can't use
as namespace labels
* build and deployed new jobsutils and misctools packages
Fri:
* Saw report of jstart not working. Fixed the problem I had caused.
* new project request triage
* rubber ducked while raynor/Pmiazga figured out how to ssh to an instance
Weekend:
* nice a quiet
Mon:
* Foundation holiday, also pretty quiet
There was no ops meeting due to the holiday. +1 again for our move to Tuesdays.
Bryan
--
Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org>
[[m:User:BDavis_(WMF)]] Manager, Cloud Services Boise, ID USA
irc: bd808 v:415.839.6885 x6855