October 2017 - Cloud-admin - lists.wikimedia.org

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 12 Oct '17

12 Oct '17

2017-10-11 20:00:02,790 INFO force is enabled 2017-10-11 20:00:02,822 INFO removing misc-project-backup 2017-10-11 20:00:02,891 INFO removing misc-project-backup 2017-10-11 20:00:03,414 INFO creating misc-project-backup at 2T 2017-10-11 20:00:04,370 INFO force is enabled 2017-10-11 20:00:04,411 INFO removing misc-snap 2017-10-11 20:00:04,465 INFO removing misc-snap 2017-10-11 20:00:04,852 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 11 Oct '17

11 Oct '17

2017-10-10 20:00:03,208 INFO force is enabled 2017-10-10 20:00:03,254 INFO removing tools-project-backup 2017-10-10 20:00:03,362 INFO removing tools-project-backup 2017-10-10 20:00:03,961 INFO creating tools-project-backup at 2T 2017-10-10 20:00:04,787 INFO force is enabled 2017-10-10 20:00:04,813 INFO removing tools-snap 2017-10-10 20:00:04,861 INFO removing tools-snap 2017-10-10 20:00:05,933 INFO creating tools-snap at 1T

1 0

Fortnightly goal planning/progress meetings
by Bryan Davis 06 Oct '17

06 Oct '17

I realized while putting our Q1 quarterly review slides together this week that I was committing all of us to work without a support plan to help make sure that things keep moving. I've setup a meeting every other week for each of the big ticket goals (Puppet4, Neutron, Dumps, Promote tools, Documentation, Labsdb). The point of these meetings will just be a quick check-in on the recently completed and upcoming work towards the goal. We can use this time to break the larger work down into more manageable chunks, talk about blockers, and celebrate the things that are getting done. Its a lot easier to reason about how much each of us can accomplish in two weeks than in 3 months. I'll be in all 6 meetings (joy!), but most of you are optional on all of them that are not your goals. Everyone is welcome in any and all of the meetings, but I don't want to make any of you show up to something that you won't find useful. I "volunteered" Quiddity to join me for the Labsdb meeting, but he can tell me to go find another person to make sure I do my project. :) Bryan -- Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org> [[m:User:BDavis_(WMF)]] Manager, Cloud Services Boise, ID USA irc: bd808 v:415.839.6885 x6855

1 0

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 05 Oct '17

05 Oct '17

2017-10-04 20:00:02,499 INFO force is enabled 2017-10-04 20:00:02,524 INFO removing misc-project-backup 2017-10-04 20:00:02,574 INFO removing misc-project-backup 2017-10-04 20:00:02,961 INFO creating misc-project-backup at 2T 2017-10-04 20:00:03,808 INFO force is enabled 2017-10-04 20:00:03,838 INFO removing misc-snap 2017-10-04 20:00:03,893 INFO removing misc-snap 2017-10-04 20:00:04,308 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 04 Oct '17

04 Oct '17

2017-10-03 20:00:02,707 INFO force is enabled 2017-10-03 20:00:02,738 INFO removing tools-project-backup 2017-10-03 20:00:02,792 INFO removing tools-project-backup 2017-10-03 20:00:03,283 INFO creating tools-project-backup at 2T 2017-10-03 20:00:03,984 INFO force is enabled 2017-10-03 20:00:04,003 INFO removing tools-snap 2017-10-03 20:00:04,089 INFO removing tools-snap 2017-10-03 20:00:05,475 INFO creating tools-snap at 1T

1 0

Ops meeting & clinic duty update 10/2
by Madhumitha Viswanathan 03 Oct '17

03 Oct '17

Ops meeting today: - Notes: https://etherpad.wikimedia.org/p/TechOps-2017-10-02 - Folks summarized the goal statuses for last quarter, and the goals for upcoming quarter - Jaime brought up decommissioning labsdb1001/3 this quarter - it's already part of our Q2 goals - https://phabricator.wikimedia.org/T142807 - The puppet goal is: = Modernize Puppet configuration management = * Upgrade PuppetDB to version 3.2 or newer * Upgrade Puppet to 4.8 or newer - I poked Chris on the labvirt1015 crash over the weekend - Guillaume likes our Social Norms :) Clinic duty updates: - Some helping on IRC - labvirt incident over the weekend - Looked into catchpoint failing tests for labsdb1001/3/5 https://phabricator.wikimedia.org/T177103 Best, -- Madhu :)

1 0

The future of Openstack Dashboard deployment (pre-ops-list draft take two)
by Andrew Bogott 03 Oct '17

03 Oct '17

-- What is the Openstack Dashboard? -- The OpenStack Dashboard is the upstream web UI for OpenStack. It's a Horizon application; Horizon is mostly python/django with a fair bit of Javascript. In the release we're currently running the js is strictly client-side but future versions seem to run some node.js on the server side as well. Our deployment includes several custom panels (which are largely self-contained) as well as a number of customizations to the stock dashboard. Because of strict API versioning, the Openstack Dashboard does not need to run the same version as the OpenStack projects that it talks to. Generally any version of the dashboard that post-dates the OpenStack versions is backwards-compatible. We're currently running the Mitaka dashboard with Liberty versions of the other Openstack Packages. We could potentially run the N or O dashboard, or even the development tip without incompatibilities. -- How is Horizon deployed now? -- 1) The upstream code is deployed via a Debian package along with its many, many dependencies. Those packages come from the Ubuntu cloud repo and are pinned to particular Openstack releases. 2) Puppet installs our various custom panels inside the debian-installed tree (which I do not regard as a hack). 3) Puppet installs an overrides.py file which monkey-patches some internal Dashboard features (this is kind of a hack but at least one explicitly implemented by upstream). 4) Puppet clobbers some upstream files with local, modified copies that live in the puppet repo but need to be tweaked for minor bug fixes and for various reasons can't be handled by overrides.py. (This is definitely a hack!) -- How would we like to deploy it? -- We'd like to deploy Horizon using Scap3 and wheels, much like Ores and Striker: 1) We'll have our own fork of the dashboard repo 2) A script on a build machine will use pip to fetch all the appropriate python dependencies for our selected Dashboard version, and pack each dependency into a wheel. 3) These wheels are committed into a deployment repo. 4) The deployment repo is pulled onto Tin, and the whole mess (with all dependencies) is pushed out into a venv on the Apache server via scap3. -- What would that get us? -- * We could deploy arbitrary source branches of the Dashboard, potentially running the upstream tip or at least picking up patches as soon as they're merged upstream. * We could maintain an internal branch of the Dashboard code in gerrit and apply local fixes or custom UI changes, managing those diffs and changes with git rather than via puppet overlays and monkey patches. * We could do this on Stretch! The Dashboard and many dependencies would be freed from reliance on the Ubuntu cloud repo (or Debian repos to be named later). Dependency chains would come directly from the python requirement files and be resolved by pip and other scripted tools. * The Dashboard deployment would no longer be strangely mingled with the puppet repo. * We'd have clear version reproducibility for the whole dependency tree -- Why wouldn't we want to do this? -- * Because we'll be running code that was pulled down by pip rather than from a debian- or ubuntu-vetted .deb. * Because change is work.

1 0

The future of Openstack Dashboard deployment (pre-ops-list draft)
by Andrew Bogott 03 Oct '17

03 Oct '17

-- What is the Openstack Dashboard? -- The OpenStack Dashboard is the upstream web UI for OpenStack. It's a Horizon application; Horizon is mostly python/django with a fair bit of Javascript. In the release we're currently running the js is strictly client-side but future versions seem to run some node.js on the server side as well. Our deployment includes several custom panels (which are largely self-contained) as well as a number of customizations to the stock dashboard. Because of strict API versioning, the Openstack Dashboard does not need to run the same version as the OpenStack projects that it talks to. Generally any version of the dashboard that post-dates the OpenStack versions is backwards-compatible. We're currently running the Mitaka dashboard with Liberty nova, but could potentially run the N or O dashboard, or even the development tip without incompatibilities. -- How is Horizon deployed now? -- 1) The upstream code is deployed a Debian package along with its many, many dependencies. Those packages come from the Ubuntu cloud repo and are pinned to particular Openstack releases. 2) Puppet installs our various custom panels inside the debian-installed tree (which I do not regard as a hack). 3) Puppet installs an overrides.py file which monkey-patches some internal Dashboard features (this is kind of a hack but at least one explicitly implemented by upstream). 4) Puppet clobbers some upstream files with local, modified copies that live in the puppet repo but need to be tweaked for minor bug fixes and for various reasons can't be handled by overrides.py. (This is definitely a hack!) -- How would we like to deploy it? -- We'd like to deploy Horizon using Scap3 and wheels: 1) We'll have our own fork of the dashboard repo 2) A script on a build machine will use pip to fetch all the appropriate python dependencies for our selected Dashboard version, and pack each dependency into a wheel. 3) These wheels are committed into a deployment repo. 4) The deployment repo is pulled onto Tin, and the whole mess (with all dependencies) is pushed out into a venv on the Apache server via scap3. -- What would that get us? -- * We could deploy arbitrary source branches of the Dashboard, potentially running the upstream tip or at least picking up patches as soon as they're merged upstream * We could maintain an internal branch of the Dashboard code in gerrit and apply local fixes or UI hacks, managing those diffs and changes with git rather than via method 3) and 4) above * The Dashboard and many dependencies would be freed from reliance on the Ubuntu cloud repo (or Debian repos to be named later). Dependency chains would come directly from the python requirement files and be resolved by pip and other scripted tools. * The Dashboard deployment would no longer be strangely mingled with the puppet repo. * We'd have clear version reproducibility for the whole dependency tree -- Why wouldn't we want to do this? -- * Because we'll be running code that was pulled down by pip rather than from a debian- or ubuntu-vetted .deb. * Because change is work.

1 0

No brogrammers: Practical tips for writing inclusive job ads
by Chase Pettet 02 Oct '17

02 Oct '17

https://blog.hostedgraphite.com/2016/04/26/no-brogrammers-practical-tips-fo…

1 0

There was just a VM outage, now resolved
by Andrew Bogott 02 Oct '17

02 Oct '17

Hello! On Friday I migrated some of your VMs to a new virt host, labvirt1015. Early today that host suffered a hardware failure, crashed, and was down for quite some time. No data was lost; your VMs have been migrated to a new host (labvirt1017) and are either back up already or will be back up and running in a few hours. Each was halted and rebooted abruptly, though, so you might want to have a look and make sure they're acting as you'd expect. For more details: https://wikitech.wikimedia.org/wiki/Incident_documentation/20171001-labvirt… Sorry for the downtime! -Andrew

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-admin October 2017