Most of the upgrade is complete; however the upgrade has caused an
unexpected partial network interruption. Work on this issue is ongoing,
I'll send another email when the issue is resolved.
-Andrew
We'll be upgrading the cloud services OpenStack install on Monday,
beginning at 06:00 UTC
The entire upgrade process may take an hour or two. Early on in the
process, Horizon (and associated OpenStack APIs) will be disabled
(probably for 20 to 30 minutes.) There may also be brief network
interruptions during the upgrade.
Toolforge and existing VMs should be largely unaffected apart from
possible network hiccups.
This is the second upgrade since October; we are trying to speed up our
upgrade schedule in order to catch up with the upstream releases. With
luck there will be further such upgrades in the coming months.
- Andrew + the WMCS team
Reminder: this maintenance is happening today, starting in about 45
minutes.
We'll be upgrading the cloud services OpenStack install on Monday,
beginning at 06:00 UTC
The entire upgrade process may take an hour or two. Early on in the
process, Horizon (and associated OpenStack APIs) will be disabled
(probably for 20 to 30 minutes.) There may also be brief network
interruptions during the upgrade.
Toolforge and existing VMs should be largely unaffected apart from
possible network hiccups.
This is the second upgrade since October; we are trying to speed up our
upgrade schedule in order to catch up with the upstream releases. With
luck there will be further such upgrades in the coming months.
- Andrew + the WMCS team
We'll be upgrading the cloud services OpenStack install on Monday,
beginning at 06:00 UTC
The entire upgrade process may take an hour or two. Early on in the
process, Horizon (and associated OpenStack APIs) will be disabled
(probably for 20 to 30 minutes.) There may also be brief network
interruptions during the upgrade.
Toolforge and existing VMs should be largely unaffected apart from
possible network hiccups.
This is the second upgrade since October; we are trying to speed up our
upgrade schedule in order to catch up with the upstream releases. With
luck there will be further such upgrades in the coming months.
- Andrew + the WMCS team
Every year or so the Cloud Services team tries to identify and clean up
unused projects and VMs. We do this via an opt-in process: anyone can
mark a project as 'in use,' and that project will be preserved for
another year.
I've created a wiki page the lists all existing projects, here:
https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2019_Purge
If you are a VPS user, please visit that page and mark any projects that
you use as {{Used}}. Note that it's not necessary for you to be a
project admin to mark something -- if you know that you're currently
using a resource and want to keep using it, go ahead and mark it
accordingly. If you /are/ a project admin, please take a moment to mark
which VMs are or aren't used in your projects.
When December arrives, I will shut down and begin the process of
reclaiming resources from unused projects.
If you think you use a VPS project but aren't sure which, I encourage
you to poke around on https://tools.wmflabs.org/openstack-browser/ to
see what looks familiar. Worst case, just email
cloud(a)lists.wikimedia.org with a description of your use case and we'll
sort it out there.
Exclusive toolforge users are free to ignore this task.
Thank you!
-Andrew and WMCS team
I'll be upgrading our Powerdns install on Wednesday. Because our system
is redundant and I'll only upgrade one node at a time, I don't expect
any service interruption. As always, though, unexpected mishaps may
cause brief service interruptions. Most likely these interruptions
would be confined to new VM creation, but in the worst case there might
be short periods of total DNS failure.
I plan to start the work at around 17:00 UTC (that's 09:00 in San
Francisco) and the total upgrade will take an hour or two.
Hi there!
Next Monday 2019-10-28 @ 14:30 UTC we will do a maintenance operation on
Toolforge which consists in rebuilding the main front proxy [0] used to serve
webservices. We expect this to be done within a 30 minutes window.
The operation consists on replacing the old virtual machines supporting the
proxy (currently running Debian Jessie) with more modern instances running
Debian Buster. Both Grid/Kubernetes backends are affected by this change. We
don't expect a lot of service downtime, but there is a key point in the
operation which is migrating data stored in Redis which can be tricky. The o
Examples of things affected by this change:
* Browsing Toolforge webservices
* Browsing to https://tools.wmflabs.org/<toolname>
* Browsing to https://tools.wmflabs.org/admin/ (Toolforge landing page)
* Browsing PAWS (to some extent, since it shares part of the toolforge proxy)
Example of things not affected by this change:
* webservices backend operations
* SSH bastions
* grid queues, grid jobs
* wiki-replicas, toolsdb
* other CloudVPS projects
regards.
[0] https://phabricator.wikimedia.org/T235627
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
With a redundant power supply upgrade going on this week in the datacenter that could affect the VM that Toolsdb runs on, we anticipate a brief outage Thursday 10/24 @11am UTC of the mysql service to protect data in case anything goes wrong. This may require a restart of a tool to reconnect to the database. We do not anticipate any worse disruptions, but if there is any disruption beyond what is planned, a failover may be necessary, which will not include the non-replicated tables mentioned here https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#ToolsDB_Backups… <https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#ToolsDB_Backups…>
The maintenance requiring this notice and action is detailed here https://phabricator.wikimedia.org/T227540 <https://phabricator.wikimedia.org/T227540>. The VM resides on the cloudvirt1019 hypervisor, which is why it is in scope.
We sincerely apologize for the short notice.
Brooke Storm
Senior SRE
Wikimedia Cloud Services
bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org>
IRC: bstorm_
Effective immediately, Toolforge's webservices (re-)started by the
webservice
command will no longer produce a $HOME/access.log file by default.
This feature can easily be re-enabled if required for your tool. To do so,
please
follow the instructions posted at https://w.wiki/9go
Since not everyone requires the access.log feature, we have decided that it
makes more sense to have it disabled by default. We believe that this change
will improve the overall Toolforge experience. Not only we can free up
disk spaces but also the CPU cycle taken up by the web servers to produce
the
access.log files.
If you see odd behaviour when starting or restarting a webservice that looks
like it could be related to this change please let myself or one of the
Toolforge admins know by either filing a Phabricator bug report or for
faster
response joining the #wikimedia-cloud IRC channel on Freenode and sending a
"!help ...." message to the channel explaining your issue.
Hieu Pham - on behalf of the Toolforge admin team