The hardware that hosts osmdb.eqiad.wmnet is long past its end of life
and will be shut down on February 12th. WMF staff do not plan to support
that database after that date, and the domain will be shut down.
I am pretty sure that we have already made arrangements with all current
users of the service, but I'm sending this email out of an abundance of
caution. If you think you are using it, please chime in on the
associated phabricator ticket[0] so that we know you exist! There is a
volunteer-maintained replacement that you should be able to switch to
with a minimum of effort.
Thanks for reading!
-Andrew + the WMCS team
[0] https://phabricator.wikimedia.org/T323159
PAWS will be upgrading to k8s 1.22 on 2023-01-31
If you were running a workload at that time it will need to be restarted.
--
*Vivian Rook (They/Them)*
Site Reliability Engineer
Wikimedia Foundation <https://wikimediafoundation.org/>
Hi there,
The Toolforge jobs framework just got upgraded with a few new features:
* support for custom logs
* support for job failure retry policy
* new behavior with job image listing
* some initial validation of YAML files
The documentation should be mostly up-to-date in wikitech:
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework
You can stop reading here unless you want more details :-)
The custom log files feature will allow you do things like:
* using a custom directory to store log files
* merging stdout/stderr logs together into a single file
* ignoring one of the two log streams
The job retry policy allows to instruct the computing engine to restart jobs
that failed, up to 5 times.
Job images are now listed in a different format, and deprecated images are
hidden by default, to encourage usage of newer ones.
Regarding the YAML validation, the toolforge-jobs utility will now emit a
warning if some key is unknown. We plan to make this more robust in the future,
also providing a schema file.
We don't usually announce upgrades, but this one in particular contained much
awaited features. This is the result of hard work by several folks, in
particular Taavi (community member) and Raymond (WMF contractor).
Happy `toolforging`. Regards.
--
Arturo Borrero Gonzalez
Senior SRE / Wikimedia Cloud Services
Wikimedia Foundation
I will be upgrading the cloud-vps openstack install on Monday afternoon
my time (beginning around 18:00 UTC). Here's what to expect:
- Intermittent Horizon and API downtime (maybe an hour or two total)
- Inability to schedule new VMs (also for an hour or two)
- Some mild Horizon dashboard changes as I'll also be upgrading the
dashboards to version 'Zen'.
Toolforge users will be unaffected by this outage. Existing, running
services and VMs on cloud-vps should also be unaffected.
-Andrew + the WMCS team
Hello cloud-vps users!
It's time for our annual cleanup of unused projects and resources. Every
year or so the Cloud Services team tries to identify and clean up unused
projects and VMs. We do this via an opt-in process: anyone can mark a
project as 'in use,' and that project will be preserved for another year.
I've created a wiki page that lists all existing projects, here:
https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge
If you are a VPS user, please visit that page and mark any projects that
you use as {{Used}}. Note that it's not necessary for you to be a
project admin to mark something -- if you know that you're currently
using a resource and want to keep using it, go ahead and mark it
accordingly. If you /are/ a project admin, please take a moment to mark
which VMs are or aren't used in your projects.
When February arrives, I will shut down and begin the process of
reclaiming resources from unused projects.
If you think you use a VPS project but aren't sure which, I encourage
you to poke around on https://tools.wmflabs.org/openstack-browser/ to
see what looks familiar. Worst case, just email
cloud(a)lists.wikimedia.org with a description of your use case and we'll
sort it out there.
Exclusive toolforge users are free to ignore this email.
Thank you!
-Andrew and the WMCS team
Hi there,
the Toolforge jobs service [0] (the one you would use via the `toolforge-jobs`
command line interface) will have a brief maintenance today 2023-01-10 @ 11:30
UTC (in about 15 minutes).
We need to restart the API service and it will be down for a couple of minutes
(perhaps even less).
During that time, using the toolforge-jobs command line interface will most
likely fail.
regards.
[0] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework
--
Arturo Borrero Gonzalez
Senior SRE / Wikimedia Cloud Services
Wikimedia Foundation
On Tuesday 2023-01-17 PAWS will be moving k8s clusters.
As a result any running workloads or active sessions will stop and need to
be restarted.
https://phabricator.wikimedia.org/T326554
--
*Vivian Rook (They/Them)*
Site Reliability Engineer
Wikimedia Foundation <https://wikimediafoundation.org/>