Hi there!
Today 2020-12-10 @ 15:30 UTC we will perform an upgrade of the Toolforge
kubernetes cluster [0].
We don't expect any major disruption of the service, but we detected in past
upgrades that some components might be restarted, causing brief interruptions of
network flows.
Given the amount of worker nodes we have, more than 50, the operation will take
us at least a couple of hours.
Tools maintainers: you don't have to do anything during this operation, but if
you detect anything weird please contact us either in the phabricator task [0],
in the IRC channel #wikimedia-cloud or in the cloud(a)lists.wikimedia.org [1]
mailing list.
regards.
[0] https://phabricator.wikimedia.org/T263284
[1] https://lists.wikimedia.org/mailman/listinfo/cloud
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
Hi there,
we are about to upgrade the kubernetes version that runs PAWS, from 1.6 to 1.17.
We don't expect any interruptions major on the service, perhaps only some
hiccups when pods are restarted/rescheduled.
More information is available in this phabricator ticket:
https://phabricator.wikimedia.org/T268669
The operation may take something between 30 minutes and 1 hours, and we are
starting soon after I finish sending this email.
Please, ping us if you see anything wrong.
regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
Hi!,
There will be a general CloudVPS network maintenance on 2020-10-09 @ 12:30 UTC.
The operation window will last for 1h. During the operation, all cloud services
will be inaccessible or intermittently down.
This operation affects all CloudVPS projects, including Toolforge, PAWS and
Quarry. Services running in the cloud might fail to contact external entities,
and connections to ToolsDB, NFS, wiki-replicas or LDAP will be affected as well.
The operation we are doing today is a followup to what we did two weeks ago [0],
and involves changing the IP addressing of the network that connects the
CloudVPS network to the internet.
Sorry for the short notice, we couldn't avoid scheduling this to today.
regards.
[0] https://phabricator.wikimedia.org/T265288
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
Hi!
There will be a general CloudVPS network maintenance on 20202-10-29, from 16:00
UTC to 17:00 UTC.
During the operation window, all cloud services might be intermittently down,
inaccessible.
This operation affects all CloudVPS projects, including Toolforge, PAWS and
Quarry. Services running in the cloud might fail to contact external entities,
and connections to ToolsDB, NFS, wiki-replicas or LDAP might be affected as well.
In the best case scenario, the changes (and downtime) will be barely noticed.
The maintenance consist on introducing new hardware equipment in to the CloudVPS
edge network. You can find additional details in Phabricator [0].
regards.
[0] https://phabricator.wikimedia.org/T265288
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
Hi there,
we need to perform some unscheduled keystone maintenance right now.
Authentication to some cloud services, in particular Horizon, might be
interrupted during this maintenance period. We expect such maintenance to don't
last more than 1h.
regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
Hi there!
wanted to share a big THANKS YOU to both Alex Monk (Krenair) and Andrew Bogott
for the amazing work they have been doing to support the codfw1dev openstack
deployment, including deploying an in-cloud puppetmaster to get it even closer
to being a full mirror of our live openstack deployment eqiad1. Note working in
getting the codfw1dev deployment in shape is something we don't usually set an
explicit goal on, even though, from my point of view, is vital for us to have a
proper testing environment.
Traditionally I've been using codfw1dev only for testing non-virtual things,
like openstack services, etc. With the current status, you can even test virtual
stuff (running in VM instances) with an experience very close to eqiad1.
This week I've been using the codfw1dev deployment for testing stuff, and for
the first time since I can remember, the experience was... mostly magically
smooth out of the box.
Now the challenge for myself (and possibly others) is to remember to test stuff
there first :-)
I case you want to try this yourself, I left here some hints on how to use the
deployment:
https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Testing_deployme…
regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
Hi!
We are happy to announce the new domain 'toolforge.org' is now ready to be
adopted by our Toolforge community.
There is a lot of information related to this change in a wikitech page we have
for this:
https://wikitech.wikimedia.org/wiki/News/Toolforge.org
The most important change you will see happening is a new domain/scheme for
Toolforge-hosted webservices:
* from https://tools.wmflabs.org/<toolname>/
* to https://<toolname>.toolforge.org/
A live example of this change can be found in our internal openstack-browser
webservice tool:
* legacy URL: https://tools.wmflabs.org/openstack-browser/
* new URL: https://openstack-browser.toolforge.org
This domain change is something we have been working on for months previous to
this announcement. Part of our work has been to ensure we have a smooth
transition from the old domain (and URL scheme) to the new canonical one.
However, we acknowledge the ride might be bumpy for some folks, due to technical
challenges or cases we didn't consider when planning this migration. Please
reach out intermediately if you find any limitation or failure anywhere related
to this change. The wikitech page also contains a section with information for
common problems.
You can check now if your webservice needs any specific change by creating a
temporal redirection to the new canonical URL:
$ webservice --canonical --backend=kubernetes start [..]
$ webservice --canonical --backend=gridengine start [..]
The --canonical switch will create a temporal redirect that you can turn on/off.
Please use this to check how your webservice behaves with the new domain/URL
scheme. If you start the webservice without --canonical, the temporal redirect
will be removed.
We aim to introduce permanent redirects for the legacy URLs on 2020-06-15. We
expect to keep serving legacy URLs forever, by means of redirections to the new
URLs. More information on the redirections can also be found in the wikitech page.
The toolforge.org domain is finally here! <3
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
Hi,
we just enabled email ratelimiting in our MTA server [0] in Toolforge.
Please, report any problem or issue you may find related to this.
The current limit is 100 messages per hour per sender address. We may tune the
value as we observe the behavior of the system and the users.
regards.
[0] https://en.wikipedia.org/wiki/Message_transfer_agent
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
Hi there!
We just deployed tesseract-ocr v4.1.1 in the Toolforge grid.
The context of this update is the phabricator task T247422 [0].
Please report any issue you may find.
regards!
[0] https://phabricator.wikimedia.org/T247422
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation