Hello cloud-vps users!
It's time for our annual cleanup of unused projects and resources. Every
year or so the Cloud Services team tries to identify and clean up unused
projects and VMs. We do this via an opt-in process: anyone can mark a
project as 'in use,' and that project will be preserved for another year.
I've created a wiki page that lists all existing projects, here:
https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge
If you are a VPS user, please visit that page and mark any projects that
you use as {{Used}}. Note that it's not necessary for you to be a
project admin to mark something -- if you know that you're currently
using a resource and want to keep using it, go ahead and mark it
accordingly. If you /are/ a project admin, please take a moment to mark
which VMs are or aren't used in your projects.
When February arrives, I will shut down and begin the process of
reclaiming resources from unused projects.
If you think you use a VPS project but aren't sure which, I encourage
you to poke around on https://tools.wmflabs.org/openstack-browser/ to
see what looks familiar. Worst case, just email
cloud(a)lists.wikimedia.org with a description of your use case and we'll
sort it out there.
Exclusive toolforge users are free to ignore this email.
Thank you!
-Andrew and the WMCS team
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…
The hardware that hosts osmdb.eqiad.wmnet is long past its end of life
and will be shut down on February 12th. WMF staff do not plan to support
that database after that date, and the domain will be shut down.
I am pretty sure that we have already made arrangements with all current
users of the service, but I'm sending this email out of an abundance of
caution. If you think you are using it, please chime in on the
associated phabricator ticket[0] so that we know you exist! There is a
volunteer-maintained replacement that you should be able to switch to
with a minimum of effort.
Thanks for reading!
-Andrew + the WMCS team
[0] https://phabricator.wikimedia.org/T323159
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…
PAWS will be upgrading to k8s 1.22 on 2023-01-31
If you were running a workload at that time it will need to be restarted.
--
*Vivian Rook (They/Them)*
Site Reliability Engineer
Wikimedia Foundation <https://wikimediafoundation.org/>
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…
I just discovered that some code I've been working on, which passes all its unit tests against sqllite, fails when I run it against
tools.db.svc.wikimedia.cloud. It's apparently due differences in the two drivers (see for example, this bug <https://github.com/sqlalchemy/alembic/issues/912>). The error I get (using SQL Alchemy 2.0.0rc3) is:
> sqlalchemy.exc.CompileError: (in table 'bot_log', column 'title'): VARCHAR requires a length on dialect mysql
So, I've got two questions:
1) Has anybody seen this problem before and figured out how to work around it? I suppose I could just declare an explicit length for that column, but what part of VARCHAR did they not understand?
2) Is there a scratch database instance I can run unit tests against? I'd rather not do this kind of testing against anything that's in production, to ensure I don't accidentally cause any damage.
Hi there,
The Toolforge jobs framework just got upgraded with a few new features:
* support for custom logs
* support for job failure retry policy
* new behavior with job image listing
* some initial validation of YAML files
The documentation should be mostly up-to-date in wikitech:
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework
You can stop reading here unless you want more details :-)
The custom log files feature will allow you do things like:
* using a custom directory to store log files
* merging stdout/stderr logs together into a single file
* ignoring one of the two log streams
The job retry policy allows to instruct the computing engine to restart jobs
that failed, up to 5 times.
Job images are now listed in a different format, and deprecated images are
hidden by default, to encourage usage of newer ones.
Regarding the YAML validation, the toolforge-jobs utility will now emit a
warning if some key is unknown. We plan to make this more robust in the future,
also providing a schema file.
We don't usually announce upgrades, but this one in particular contained much
awaited features. This is the result of hard work by several folks, in
particular Taavi (community member) and Raymond (WMF contractor).
Happy `toolforging`. Regards.
--
Arturo Borrero Gonzalez
Senior SRE / Wikimedia Cloud Services
Wikimedia Foundation
I will be upgrading the cloud-vps openstack install on Monday afternoon
my time (beginning around 18:00 UTC). Here's what to expect:
- Intermittent Horizon and API downtime (maybe an hour or two total)
- Inability to schedule new VMs (also for an hour or two)
- Some mild Horizon dashboard changes as I'll also be upgrading the
dashboards to version 'Zen'.
Toolforge users will be unaffected by this outage. Existing, running
services and VMs on cloud-vps should also be unaffected.
-Andrew + the WMCS team
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…
Hi,
I am working on the migration of ToolsDB [1] from the existing VMs to
new VMs with more recent versions of Debian and MariaDB.
The database will be set to read-only later today, and potentially
again later this week if the first backup attempt fails. My goal is to
keep the read-only time to a maximum of 30 minutes. If mariabackup is
unable to complete the backup after 30 minutes of read-only lock, I
will cancel it and prepare a different backup plan.
Apologies for any inconvenience, please ping me in IRC
(#wikimedia-cloud-admin) if your tool is having issues reading or
writing to the database.
This work is tracked at https://phabricator.wikimedia.org/T301949
[1] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database
--
Francesco Negri (he/him) -- IRC: dhinus
Site Reliability Engineer, Cloud Services team
Wikimedia Foundation
Hi there,
the Toolforge jobs service [0] (the one you would use via the `toolforge-jobs`
command line interface) will have a brief maintenance today 2023-01-10 @ 11:30
UTC (in about 15 minutes).
We need to restart the API service and it will be down for a couple of minutes
(perhaps even less).
During that time, using the toolforge-jobs command line interface will most
likely fail.
regards.
[0] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework
--
Arturo Borrero Gonzalez
Senior SRE / Wikimedia Cloud Services
Wikimedia Foundation
Hello all,
As you may have heard, the Wikimedia Hackathon
<https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2023> is back in person
on May 19-21 in Athens, Greece. We are now opening the registration form
that also includes an optional scholarship application form open until
January 14th.
Registration is required to attend the onsite Wikimedia Hackathon in
Athens. The registration form remains open until we reach the capacity of
the venue (around 220 people). The event is free of charge - participants
are expected to book their travel arrangements individually, unless they
are granted a scholarship.
You can register here by getting yourself a (free) ticket for the hackathon!
<https://pretix.eu/wikimedia/wmhack2023/> (see also the detailed
instructions
<https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2023/Participate#Registr…>
)
In order to support the technical community and participants from diverse
backgrounds, the Wikimedia Foundation provides scholarships that cover
travel and accommodation for a selected number of technical contributors.
The application form is part of the registration process, and is open
until January
14th. You can find more information about the scholarship process,
committee and criteria on the related page
<https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2023/Participate#Scholar…>
.
The registration and scholarship application form runs with Pretix, an open
source third-party service, which may subject it to additional terms. For
more information on privacy and data-handling, see the privacy statement
<https://foundation.wikimedia.org/wiki/Legal:Wikimedia_Hackathon_2023_regist…>
.
If you’d like to connect with others on plans for the hackathon, the talk
page <https://www.mediawiki.org/wiki/Talk:Wikimedia_Hackathon_2023/Connect>
and the other channels listed here
<https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2023/Connect> are good
places to get started!
If you have any questions or issues with the registration form and/or the
scholarship application, feel free to reach out to the organization team on
the talk page or at hackathon(a)wikimedia.org.
For the organization team,
--
*Léa Lacroix *(she/her)
Hackathon Event Coordinator
Wikimedia Foundation <https://wikimediafoundation.org/>