2017-10-11 20:00:02,790 INFO force is enabled
2017-10-11 20:00:02,822 INFO removing misc-project-backup
2017-10-11 20:00:02,891 INFO removing misc-project-backup
2017-10-11 20:00:03,414 INFO creating misc-project-backup at 2T
2017-10-11 20:00:04,370 INFO force is enabled
2017-10-11 20:00:04,411 INFO removing misc-snap
2017-10-11 20:00:04,465 INFO removing misc-snap
2017-10-11 20:00:04,852 INFO creating misc-snap at 1T
2017-10-10 20:00:03,208 INFO force is enabled
2017-10-10 20:00:03,254 INFO removing tools-project-backup
2017-10-10 20:00:03,362 INFO removing tools-project-backup
2017-10-10 20:00:03,961 INFO creating tools-project-backup at 2T
2017-10-10 20:00:04,787 INFO force is enabled
2017-10-10 20:00:04,813 INFO removing tools-snap
2017-10-10 20:00:04,861 INFO removing tools-snap
2017-10-10 20:00:05,933 INFO creating tools-snap at 1T
I realized while putting our Q1 quarterly review slides together this
week that I was committing all of us to work without a support plan to
help make sure that things keep moving. I've setup a meeting every
other week for each of the big ticket goals (Puppet4, Neutron, Dumps,
Promote tools, Documentation, Labsdb).
The point of these meetings will just be a quick check-in on the
recently completed and upcoming work towards the goal. We can use this
time to break the larger work down into more manageable chunks, talk
about blockers, and celebrate the things that are getting done. Its a
lot easier to reason about how much each of us can accomplish in two
weeks than in 3 months.
I'll be in all 6 meetings (joy!), but most of you are optional on all
of them that are not your goals. Everyone is welcome in any and all of
the meetings, but I don't want to make any of you show up to something
that you won't find useful. I "volunteered" Quiddity to join me for
the Labsdb meeting, but he can tell me to go find another person to
make sure I do my project. :)
Bryan
--
Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org>
[[m:User:BDavis_(WMF)]] Manager, Cloud Services Boise, ID USA
irc: bd808 v:415.839.6885 x6855
2017-10-04 20:00:02,499 INFO force is enabled
2017-10-04 20:00:02,524 INFO removing misc-project-backup
2017-10-04 20:00:02,574 INFO removing misc-project-backup
2017-10-04 20:00:02,961 INFO creating misc-project-backup at 2T
2017-10-04 20:00:03,808 INFO force is enabled
2017-10-04 20:00:03,838 INFO removing misc-snap
2017-10-04 20:00:03,893 INFO removing misc-snap
2017-10-04 20:00:04,308 INFO creating misc-snap at 1T
2017-10-03 20:00:02,707 INFO force is enabled
2017-10-03 20:00:02,738 INFO removing tools-project-backup
2017-10-03 20:00:02,792 INFO removing tools-project-backup
2017-10-03 20:00:03,283 INFO creating tools-project-backup at 2T
2017-10-03 20:00:03,984 INFO force is enabled
2017-10-03 20:00:04,003 INFO removing tools-snap
2017-10-03 20:00:04,089 INFO removing tools-snap
2017-10-03 20:00:05,475 INFO creating tools-snap at 1T
Ops meeting today:
- Notes: https://etherpad.wikimedia.org/p/TechOps-2017-10-02
- Folks summarized the goal statuses for last quarter, and the goals for
upcoming quarter
- Jaime brought up decommissioning labsdb1001/3 this quarter - it's already
part of our Q2 goals - https://phabricator.wikimedia.org/T142807
- The puppet goal is:
= Modernize Puppet configuration management =
* Upgrade PuppetDB to version 3.2 or newer
* Upgrade Puppet to 4.8 or newer
- I poked Chris on the labvirt1015 crash over the weekend
- Guillaume likes our Social Norms :)
Clinic duty updates:
- Some helping on IRC
- labvirt incident over the weekend
- Looked into catchpoint failing tests for labsdb1001/3/5
https://phabricator.wikimedia.org/T177103
Best,
--
Madhu :)
-- What is the Openstack Dashboard? --
The OpenStack Dashboard is the upstream web UI for OpenStack. It's a
Horizon application; Horizon is mostly python/django with a fair bit of
Javascript. In the release we're currently running the js is strictly
client-side but future versions seem to run some node.js on the server
side as well.
Our deployment includes several custom panels (which are largely
self-contained) as well as a number of customizations to the stock
dashboard.
Because of strict API versioning, the Openstack Dashboard does not need
to run the same version as the OpenStack projects that it talks to.
Generally any version of the dashboard that post-dates the OpenStack
versions is backwards-compatible. We're currently running the Mitaka
dashboard with Liberty versions of the other Openstack Packages. We
could potentially run the N or O dashboard, or even the development tip
without incompatibilities.
-- How is Horizon deployed now? --
1) The upstream code is deployed via a Debian package along with its
many, many dependencies. Those packages come from the Ubuntu cloud repo
and are pinned to particular Openstack releases.
2) Puppet installs our various custom panels inside the debian-installed
tree (which I do not regard as a hack).
3) Puppet installs an overrides.py file which monkey-patches some
internal Dashboard features (this is kind of a hack but at least one
explicitly implemented by upstream).
4) Puppet clobbers some upstream files with local, modified copies that
live in the puppet repo but need to be tweaked for minor bug fixes and
for various reasons can't be handled by overrides.py. (This is
definitely a hack!)
-- How would we like to deploy it? --
We'd like to deploy Horizon using Scap3 and wheels, much like Ores and
Striker:
1) We'll have our own fork of the dashboard repo
2) A script on a build machine will use pip to fetch all the appropriate
python dependencies for our selected Dashboard version, and pack each
dependency into a wheel.
3) These wheels are committed into a deployment repo.
4) The deployment repo is pulled onto Tin, and the whole mess (with all
dependencies) is pushed out into a venv on the Apache server via scap3.
-- What would that get us? --
* We could deploy arbitrary source branches of the Dashboard,
potentially running the upstream tip or at least picking up patches as
soon as they're merged upstream.
* We could maintain an internal branch of the Dashboard code in gerrit
and apply local fixes or custom UI changes, managing those diffs and
changes with git rather than via puppet overlays and monkey patches.
* We could do this on Stretch! The Dashboard and many dependencies
would be freed from reliance on the Ubuntu cloud repo (or Debian repos
to be named later). Dependency chains would come directly from the
python requirement files and be resolved by pip and other scripted tools.
* The Dashboard deployment would no longer be strangely mingled with the
puppet repo.
* We'd have clear version reproducibility for the whole dependency tree
-- Why wouldn't we want to do this? --
* Because we'll be running code that was pulled down by pip rather than
from a debian- or ubuntu-vetted .deb.
* Because change is work.
-- What is the Openstack Dashboard? --
The OpenStack Dashboard is the upstream web UI for OpenStack. It's a
Horizon application; Horizon is mostly python/django with a fair bit of
Javascript. In the release we're currently running the js is strictly
client-side but future versions seem to run some node.js on the server
side as well.
Our deployment includes several custom panels (which are largely
self-contained) as well as a number of customizations to the stock
dashboard.
Because of strict API versioning, the Openstack Dashboard does not need
to run the same version as the OpenStack projects that it talks to.
Generally any version of the dashboard that post-dates the OpenStack
versions is backwards-compatible. We're currently running the Mitaka
dashboard with Liberty nova, but could potentially run the N or O
dashboard, or even the development tip without incompatibilities.
-- How is Horizon deployed now? --
1) The upstream code is deployed a Debian package along with its many,
many dependencies. Those packages come from the Ubuntu cloud repo and
are pinned to particular Openstack releases.
2) Puppet installs our various custom panels inside the debian-installed
tree (which I do not regard as a hack).
3) Puppet installs an overrides.py file which monkey-patches some
internal Dashboard features (this is kind of a hack but at least one
explicitly implemented by upstream).
4) Puppet clobbers some upstream files with local, modified copies that
live in the puppet repo but need to be tweaked for minor bug fixes and
for various reasons can't be handled by overrides.py. (This is
definitely a hack!)
-- How would we like to deploy it? --
We'd like to deploy Horizon using Scap3 and wheels:
1) We'll have our own fork of the dashboard repo
2) A script on a build machine will use pip to fetch all the appropriate
python dependencies for our selected Dashboard version, and pack each
dependency into a wheel.
3) These wheels are committed into a deployment repo.
4) The deployment repo is pulled onto Tin, and the whole mess (with all
dependencies) is pushed out into a venv on the Apache server via scap3.
-- What would that get us? --
* We could deploy arbitrary source branches of the Dashboard,
potentially running the upstream tip or at least picking up patches as
soon as they're merged upstream
* We could maintain an internal branch of the Dashboard code in gerrit
and apply local fixes or UI hacks, managing those diffs and changes with
git rather than via method 3) and 4) above
* The Dashboard and many dependencies would be freed from reliance on
the Ubuntu cloud repo (or Debian repos to be named later). Dependency
chains would come directly from the python requirement files and be
resolved by pip and other scripted tools.
* The Dashboard deployment would no longer be strangely mingled with the
puppet repo.
* We'd have clear version reproducibility for the whole dependency tree
-- Why wouldn't we want to do this? --
* Because we'll be running code that was pulled down by pip rather than
from a debian- or ubuntu-vetted .deb.
* Because change is work.
Hello!
On Friday I migrated some of your VMs to a new virt host, labvirt1015.
Early today that host suffered a hardware failure, crashed, and was down
for quite some time.
No data was lost; your VMs have been migrated to a new host
(labvirt1017) and are either back up already or will be back up and
running in a few hours. Each was halted and rebooted abruptly, though,
so you might want to have a look and make sure they're acting as you'd
expect.
For more details:
https://wikitech.wikimedia.org/wiki/Incident_documentation/20171001-labvirt…
Sorry for the downtime!
-Andrew