This meeting coalesced around a few major topics:
* Why not just bring-your-own-container?
** We turn out to have fairly different ideas about what user
experiences we want to support.
** General agreement that we could use more research and/or
documentation about current and future workflows (although some of that
already exists at
https://www.mediawiki.org/wiki/Wikimedia_Cloud_Services_team/Our_audiences
** This question hinges on how sophisticated our users are or aren't.
** Komla suggests that many public cloud platforms provide simple
deployment that don't require users to understand about container or k8s
details; there's general agreement that we would also like to be like that
** Andrew thinks that push-to-deploy should be our 'main priority' and
byo doesn't really address that.
* How/When will we kill off the grid engine?
** We tend to think of this as blocked by push-to-deploy, but perhaps we
should be open to other non-blocking options (e.g. the 'one big
container' migration path)
* What to do about the Stretch->Buster migration?
** Nicholas isn't convinced that we should migrate to Buster if we're
just going to kill the grid eventually anyway. Andrew and Arturo mostly
disagree.
** Probably the migration to Buster isn't a lot of hard mental work,
just building things and throwing pre-existing switches.
** Main blocker for this is allocating time and tasking someone with
doing the work
------ RAW ETHERPAD NOTES ------
== Toolforge next steps meeting 2021-12-14 ==
The approaching deadlines (from an announcement email on 2021-09-27) is:
January 1st, 2022:
* Stretch alternatives will be available for tool migration in Toolforge
The proposed agenda follows.
=== Goals (Grid) ===
* Grid engine deprecation is blocked until users can customize container
builds. Buildpacks are intended to address this need.
* The blocker is providing equivalent runtime support for tools on the
Kubernetes cluster as the current grid engine cluster has. Kubernetes
containers are "thin" and tools will need the ability to add libraries
and binaries that are custom to them.
=== Goals (Buildpacks) ===
** Why are we doing this?
*** Allow users to customize k8s images.. Better able to allow users to
migrate off of gridengine
* Arturo asks: why not just bring-your-own-container?
** Bryan answers: because bring-your-own container means containers
without any toolforge integration (e.g. no ldap)
* What about putting that in a base layer?
** That's a build your own container approach, which is what buildpacks
is bringing
** But allowing build-your-own today is simpler than buildpacks for SREs
** Adding complexity for end users
* Building your own docker image adds more complexity for end users
** Buildpacks also limit what you can put in a docker container, so
potentially better security
**
* How does buildpacks improve security?
** If you let someone else build a docker image, it could run as a
different user and open security holes
** How could it be limited in k8s? The container runs as root on the host
** By building the container in buildpacks, we limit it
** k8s has full control over the runc runtime, so k8s could prevent user
spoofing
* Public clouds have bring your own container, so it must be possible right?
* Would like to see a list of prioritized user workflows
** What workflows are we enabling with buildpacks?
** Consider looking at
https://www.mediawiki.org/wiki/Wikimedia_Cloud_Services_team/Our_audiences
* What's the long-term vision for TF?
** Push to git repo, and it "just works"
** The heroku workflow is built on buildpacks
* What's the concern / fear about buildpacks?
** Complexity, can we find a way to simplify things?
** Lack of adoption (what if we end up the only ones using it)
* Bringing your own container isn't a regression, but it's not a
replacement for existing workflows (aka, we can't kill the grid by
simply adding bring your own container)
** Why?
** Workflow and brainstate. Users will no longer understand how to run
workflows. Running a job is much simpler than building and maintaining a
container to run a job. Real risk of losing tools
* Can we assume complex tool authors are technically capable of building
a container?
** Complexity can be easily introduced in the grid; that doesn't mean
they could build and maintain a container
** `webservice --backend=gridengine start` supports PHP, Python, and
Perl fcgi out of the box.
* Google cloud comparision -- industry seems to be using towards
containers, but users don't need to build a container or even know that
you are running in a container. Build a flask app, run a command, GCE
builds a docker container and runs it
** all you need is for the buildpack to 'detect' what runtimes you need
(it can be a file, or chekcking for a packages.lock or whatever)
* So David et la -- does buildpacks seem complex?
** Buildpacks are easy-ish. Complexity is introduced by putting
buildpacks into Toolforge
** Tekton / buildpacks PoC is easy
* Build service
** admission controller, some custom resource definitions, hardware. Any
docker registry could be used? Harbor presented some issues in running
outside of k8s
* How much engineering effort to bring that in prod? Is it possible to
bring online this year?
** Depends on resources, and how org handles things
** Yes, if we work on it? :-)
* Priority wise, a push to deploy solution is the most important /
seamless thing we could work on
** Push to deploy toolforge, before grid deprecation even?
** Can't get rid of the grid until reasonable replacement
** Why?
** Because we need to support people today
** Actually need mixed runtime environment deployable on k8s
* For example, 1 giant container that contains everything from a grid
exec node
** So grid isn't dependent on push to deploy exactly
=== Goals (Buster) ===
* Have a plan for Buster on grid engine
* Decide whether or not to have a plan for Buster on k8s (and,
optionally, have a plan)
* Decide what timeline adjustment is realistic; pick someone to
communicate this delay to the users
==== Grid Engine ====
* Why do we hate the grid?
** When jobs are being run, little isolation. Uses watcher spawned
processes + runtime hacks.
** No longer developed or supported by any upstreams
** In ~2018? looked at modern "grids" that spawned things similar, but
could be better managed. (IE, slurm)
** At the time, decided k8s was the future, and decided that slurm or
similar wasn't a good idea
* Grid is important -- what could we do this year?
** Find someone to build a buster grid
** Make a mad dash at killing the grid asap.
* Giant container is unlocked now , previously limited to 2G
** 3.1G container, needed on each k8s exec node
** Only need 1 copy on each node; shared between jobs
** When building new containers, be wary of variants when deploying new
containers. Could be N x 3.1G
** Large containers aren't performant on k8s
* Buster migration
** Most of the pieces are in place
** Build out nodes, switchover
* Why the timeline?
** Organizational timeline
** Grid is senstive to DNS changes
<discuss>
<decide>
<who>
==== k8s ====
<discuss>
<decide>
<who>
==== timeline ====
<discuss>
<decide>
<who>
* dcaro: I propose delaying any non-urgent decision until we finish with
the decision making/tech discussion email thread
** I would argue that the stretch deprecation is starting to become
urgent (although I don't have context about that email thread)
=== Current status, open projects ===
The list may be the list of open projects:
* stretch-to-buster migration final push
** including but not limited to the grid
* grid engine deprecation timeline & plans
** draft here
https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/Enhanceme…
* pending kubernetes upgrades (if any)
** yes, current clusters are at 1.20 and 1.23 is the latest release.
1.21 is simple (and I think it would be a good opportunity for someone
to learn the process) but 1.22 is complicated because it removes tons of
deprecated things
** 1.20 is still in supported, support policy is last 4 releases
* toolforge build service (buildpacks)
** currently we don't really have any visibility to package versions and
available (security) upgrades in our images or deployed image versions -
can we improve this with buildpack images or otherwise?
https://phabricator.wikimedia.org/T291908
* toolforge jobs framework
=== Next steps, prioritization ===
* what to do next, and who
=== Long term future ===
* Share your ideas of how Toolforge should look in 5 years from now
Context:
"Galera on cloudcontrol1004 going out of sync"
https://phabricator.wikimedia.org/T302146
Galera (the database backend for OpenStack) has been very unstable ever
since I upgraded the cluster to Bullseye. This is probably an issue with
a buggy version of mariadb/galera.
I'm trying an experiment: mariadb is currently stopped on
cloudcontrol1004, and puppet disabled so it won't get restarted. I want
to see if that change (a two-node cluster and/or removing the
suspected-cursed cloudcontrol1004 from the cluster) causes things to
stop breaking.
I've done my best to downtime alerts that will result from this, bug if
one leaks through please don't respond by enabling puppet on 1004 -- we
want to leave that db node switched off for now.
Other services on cloudcontrol1004 should continue to run normally.
Thanks!
-Andrew
Hi there,
Apparently linkwatcher is a key tool in fighting vandalism in the wikis,
but we cannot afford hosting it on ToolsDB any longer.
I requests your comments / suggestions.
Timeline:
==== 8< ====
== 2019-05-22 ==
Brooke detects high storage usage problem by the linkwatcher tool in
ToolsDB [0].
[0] https://phabricator.wikimedia.org/T224154
== 2019-06-26 ==
Bryan warns [1] about the linkwatcher tool potentially becoming a
disaster condition for ToolsDB.
[1] https://phabricator.wikimedia.org/T224154#5284580
== 2019-07-19 ==
After some actions are taken to move linkwatcher to its own Cloud VPS
project, activity stops and the database data remains in ToolsDB [2].
[2] https://phabricator.wikimedia.org/T227377
== 2022-02-17 ==
ToolsDB stops replicating to its secondary due to depleted storage [3].
It is discovered [4] that linkwatcher uses 1/3 of total storage of
ToolsDB, this is ~1TB out of ~3TB. Data reported in other metrics [5]
may not be accurate.
While it may not be the primary cause of the problem, the "disaster
condition" mentioned by Bryan 3 years earlier is becoming more apparent.
[3] https://phabricator.wikimedia.org/T301951
[4] https://phabricator.wikimedia.org/T301967
[5] https://tool-db-usage.toolforge.org/
==== 8< ====
regards
--
Arturo Borrero Gonzalez
Site Reliability Engineer
Wikimedia Cloud Services
Wikimedia Foundation
Hi all,
Toolforge now has a PuppetDB server! Thanks to Krenair's awesome work
that makes it easy to add a PuppetDB server to a Puppet master using
`role::puppetmaster::standalone`, I've installed tools-puppetdb-1 which
seems to be working fine. For now it's only managing SSH host keys
between hosts, but I'm sure you all can find some clever and awesome
uses for it.
In related news, there is now also a Cumin server in Toolforge and
Toolsbeta (tools-cumin-1 and toolsbeta-cumin-1, respectively). These
servers are hooked up to the per-project PuppetDB servers, allowing use
of Cumin queries like this:
$ cumin "P{P:toolforge::grid::node::web::generic}"
4 hosts will be targeted:
tools-sgewebgen-10-[1-2].tools.eqiad1.wikimedia.cloud,tools-sgewebgrid-generic-[0901-0902].tools.eqiad.wmflabs
Enjoy!
-- Taavi