Every year or so the Cloud Services team tries to identify and clean up
unused projects and VMs. We do this via an opt-in process: anyone can
mark a project as 'in use,' and that project will be preserved for
another year.
I've created a wiki page the lists all existing projects, here:
https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge
If you are a VPS user, please visit that page and mark any projects that
you use as {{Used}}. Note that it's not necessary for you to be a
project admin to mark something -- if you know that you're currently
using a resource and want to keep using it, go ahead and mark it
accordingly. If you /are/ a project admin, please take a moment to mark
which VMs are or aren't used in your projects.
When December arrives, I will shut down and begin the process of
reclaiming resources from unused projects.
If you think you use a VPS project but aren't sure which, I encourage
you to poke around on https://tools.wmflabs.org/openstack-browser/ to
see what looks familiar. Worst case, just email
cloud(a)lists.wikimedia.org with a description of your use case and we'll
sort it out there.
Exclusive toolforge users are free to ignore this task.
Thank you!
-Andrew and WMCS team
Hi!
There will be a general CloudVPS network maintenance on 20202-10-29, from 16:00
UTC to 17:00 UTC.
During the operation window, all cloud services might be intermittently down,
inaccessible.
This operation affects all CloudVPS projects, including Toolforge, PAWS and
Quarry. Services running in the cloud might fail to contact external entities,
and connections to ToolsDB, NFS, wiki-replicas or LDAP might be affected as well.
In the best case scenario, the changes (and downtime) will be barely noticed.
The maintenance consist on introducing new hardware equipment in to the CloudVPS
edge network. You can find additional details in Phabricator [0].
regards.
[0] https://phabricator.wikimedia.org/T265288
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
On Tuesday (2020-10-20) we'll be performing upgrades on the hardware
that hosts the postgres database used by the cloud-vps 'maps' project.
This will probably result in that project being unable to serve tiles
for the duration of the upgrade.
Things will be down for an hour or two, starting around 17:00 UTC.
Earlier today a vulnerability was uncovered that might have permitted
unauthorized users to manipulate project membership or create or delete
VMs within cloud-vps. That issue has since been resolved, and there is
currently no evidence that anyone exploited it.
Nevertheless, out of an abundance of caution: if you are a project
admin, please review the members and projectadmins of your project. If
you see any unknown or untrusted users or unexpected instances, remove
them and notify me directly about what you found and what you've done.
Thank you!
-Andrew + the WMCS team
Hi there,
we need to perform some unscheduled keystone maintenance right now.
Authentication to some cloud services, in particular Horizon, might be
interrupted during this maintenance period. We expect such maintenance to don't
last more than 1h.
regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation