Several people have asked on IRC and Phabricator if the deprecation of
*.wmflabs names for Cloud VPS instances means that the service names
used to connect to ToolsDB and the Wiki Replicas are changing. The
answer is no, these names are not changing yet. We will be replacing
these service names eventually, but for now they are staying in the
wmflabs pseudo domain.
In 2017 [0] we established new canonical service names for accessing
the shared database servers for the Cloud Services environment. Those
names are still the same today.
The naming convention for connecting to the Wiki Replica servers is:
"<wikidb>.{analytics,web}.db.svc.eqiad.wmflabs". The
*.web.db.svc.eqiad.wmflabs service names are intended for queries that
need a real-time response. The *.analytics.db.svc.eqiad.wmflabs
service names are intended for batch jobs and long running queries.
See the announcement from 2017 for more details [0].
The preferred service name for ToolsDB is tools.db.svc.eqiad.wmflabs.
[0]: https://phabricator.wikimedia.org/phame/post/view/70/new_wiki_replica_serve…
Bryan, on behalf of the Cloud Services team
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
Hi everyone!
The 2019 survey collected feedback from Toolforge project members and Cloud
VPS project administrators on how the services offered can be improved to
help their development and maintenance needs. It ran from 2019-11-25 to
2019-12-13, and had 108 participants.
Due to unforeseen circumstances this year, the analysis of the data and
publishing of the results suffered delays. We finally have been able to
publish the results, which you can read on wiki:
https://meta.wikimedia.org/wiki/Research:Cloud_Services_Annual_Survey/2019
Thanks to everyone who participated and provided input and comments. They
are very useful and instrumental in shaping the future and improvements for
the cloud services.
We will launch the 2020 Cloud Services survey in a couple of months
following a similar methodology.
Have a nice day!
tl;dr:
VMs created on or after September 8th will stop having .eqiad.wmflabs
domains, and be found only under .eqiad1.wikimedia.cloud
The whole story:
Currently cloud-vps VMs stand astride two worlds: wmflabs and
wikimedia.cloud. Here's the status quo:
- New VMs get three different DNS entries:
hostname.project.eqiad1.wikimedia.cloud, hostname.project.eqiad.wmflabs,
and hostname.eqiad.wmflabs [0]
- Reverse DNS lookups return hostnames under eqiad1.wikimedia.cloud
- VMs themselves believe (e.g. via hostname -f) that they're still under
eqiad.wmflabs
That hybrid system has done a good job maintaining backwards
compatibility, but it's a bit of a mess. In the interest of simplifying,
standardizing, and eliminating ever more uses of the term 'labs', we're
going to start phasing out the wmflabs domain name. Beginning on
September 8th, new VMs will no longer receive any naming associated with
.wmflabs [1].
- New VMs will get one DNS entry: hostname.project.eqiad1.wikimedia.cloud
- New VMs will continue to have a pointer DNS entry that refers to the
.wikimedia.cloud name
- New VMs will be assigned an internal hostname under .wikimedia.cloud
In order to avoid breaking existing systems, these changes will NOT be
applied retroactively to existing VMs. Old DNS entries will live on
until the VM is deleted and should be largely harmless. If, however,
you find yourself rewriting code in order to deal with VMs under both
domains (due to the change in hostname -f behavior), don't worry --
adjusting an old VM to identify as part of .wikimedia.cloud only
requires a simple change to /etc/hosts. I'll be available to make that
change for any project that chooses consistency over
backwards-compatibility.
[0]
https://phabricator.wikimedia.org/phame/post/view/191/new_names_for_everyone
[1] https://phabricator.wikimedia.org/T260614
https://toolsadmin.wikimedia.org has been updated to a new version of
Striker [0]. This is a feature release that carries some quality of
life improvements for tool maintainers and updates for recent changes
to Toolforge webservice URLs [1]:
* Allow self-service creation of Phabricator projects for Tools
* Allow tool maintainers to delete toolinfo records
* Improved explanation of toolinfo fields
* Use *.toolforge.org URLs when generating toolinfo data
The killer feature in this list is self-service Phabricator project
creation! This action is available from the details page for any tool
right under the prior option for creating Diffusion repositories.
Many, many thanks to Taavi Väänänen (User:Majavah) for writing the
code for Phabricator project creation. Even more thanks for their
patience in waiting for code review and deployment.
[0]: https://wikitech.wikimedia.org/wiki/Striker
[1]: https://wikitech.wikimedia.org/wiki/Toolsadmin.wikimedia.org/Deployments#20…
Bryan, on behalf of the Toolforge admin team
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
A change was just now made to the shared proxy system for Toolforge
which makes the proxy respond with default content for /favicon.ico
and /robots.txt when a tool's webservice returns a 404 Not Found
response for these files.
The default /favicon.ico is the same as
<https://tools-static.wmflabs.org/toolforge/favicons/favicon.ico>.
The default robots.txt denies access to all compliant web crawlers. We
decided that this "fail closed" approach would be safer than a "fail
open" telling all crawlers to crawl all tools. Any tool that does wish
to be indexed by search engines and other crawlers can serve their own
/robots.txt content. Please see <https://www.robotstxt.org/> for more
information on /robots.txt in general.
These changes fix a regression [0] in functionality caused by the
toolforge.org migration and the introduction of the 2020 Kubernetes
ingress layer. Previously the /robots.txt and /favicon.ico from the
"admin" tool were served for all tools due to the use of a shared
hostname.
[0]: https://phabricator.wikimedia.org/T251628
Bryan, on behalf of the Toolforge admin team
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
TL;DR:
* HTTP -> HTTPS redirection is live (finally!)
* Currently allowing a "POST loophole"
* "POST loophole" will be closed on 2021-02-01
Today we merged a small change [0] to the front proxy used by Cloud
VPS projects [1]. This change brings automatic HTTP -> HTTPS
redirection to the "domain proxy" service and a
Strict-Transport-Security header with a 1 day duration.
The current configuration is conservative. We will only redirect GET
and HEAD requests to HTTPS to avoid triggering bugs in the handling of
redirects during POST requests. This "POST loophole" is the same
process that we followed when converting the production wiki farm and
Toolforge to HTTPS.
When we announced similar changes for Toolforge in 2019 [2] we forgot
to set a timeline for closing the POST loophole. This time we are
wiser! We will close the POST loophole and make all HTTP requests,
regardless of the verb used, redirect to HTTPS on 2021-02-01. This 6
month transition period should give us all a chance to find and update
URLs to use https and to fix any dependent software that might break
if a redirect was sent for a POST request.
If you find issues in your projects resulting from this change, please
do let us know. The tracking task for this change is T120486 [3]. We
also provide support in the #wikimedia-cloud channel on Freenode and
via the cloud(a)lists.wikimedia.org mailing list [4].
[0]: https://gerrit.wikimedia.org/r/c/operations/puppet/+/620122/
[1]: https://wikitech.wikimedia.org/wiki/Help:Using_a_web_proxy_to_reach_Cloud_V…
[2]: https://phabricator.wikimedia.org/phame/post/view/132/migrating_tools.wmfla…
[3]: https://phabricator.wikimedia.org/T120486
[4]: https://lists.wikimedia.org/mailman/listinfo/cloud
Bryan, on behalf of the Cloud VPS admin team
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
WMCS has redesigned the PAWS cluster and built a new one in parallel. We
would like people to try things out and makes sure it's ready to go. If
you are a PAWS user and have some things you might like to try out, you
can visit https://hub.paws.wmcloud.org and give it a shot. Please create
a Phabricator task with the project tag #paws if you find any bugs or
things that don't work as well as in the old cluster. We will leave both
clusters live until August 7th (2020-08-07), when we will change
paws.wmflabs.org to a redirect to the new domain.
Please be aware that it is running on the local sqlite database for now,
and that may reduce the performance for things like launching notebooks
under load. It will be using Toolsdb like the existing PAWS cluster
after the final cut over.
For further details, please see:
https://wikitech.wikimedia.org/wiki/News/2020_PAWS_migration
--
Brooke Storm
SRE
Wikimedia Cloud Services
bstorm(a)wikimedia.org
IRC: bstorm_
Today we merged two small changes [0][1] to the front proxy for
*.toolforge.org. These changes allowed us to close a 5 year old
feature request [2] asking for Toolforge to always use TLS (HTTPS) and
to also set a strict-transport-security header (HSTS) to tell web
browser that they should *always* use TLS when talking to a Toolforge
webservice.
Most of this has been happening for some time, but the final changes
were to increase the HSTS duration to one year (technically we
advertise 31,622,400 seconds which is 366 days) and to close the "POST
loophole". The "POST loophole" was created when TLS was first enforced
on Toolforge back in January 2019 [3]. It allowed HTTP requests with
the POST verb to continue without TLS encryption. This was done
because of unspecified behavior of clients (web browsers) when
receiving an HTTP "301 Permanent Redirect" response to a POST action.
A similar exception was originally made when the Wikimedia project
wikis were switched to always require TLS encryption.
We do not expect new issues with the use of Toolforge webservices as a
result of this change, but if you find something behaving badly as a
result please report it in Phabricator using the #Toolforge project
tag or join us in the #wikimedia-cloud Freenode IRC channel to ask for
help.
[0]: https://gerrit.wikimedia.org/r/612947
[1]: https://gerrit.wikimedia.org/r/612948
[2]: https://phabricator.wikimedia.org/T102367
[3]: https://phabricator.wikimedia.org/phame/post/view/132/migrating_tools.wmfla…
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
Hi!
We are happy to announce the new domain 'toolforge.org' is now ready to be
adopted by our Toolforge community.
There is a lot of information related to this change in a wikitech page we have
for this:
https://wikitech.wikimedia.org/wiki/News/Toolforge.org
The most important change you will see happening is a new domain/scheme for
Toolforge-hosted webservices:
* from https://tools.wmflabs.org/<toolname>/
* to https://<toolname>.toolforge.org/
A live example of this change can be found in our internal openstack-browser
webservice tool:
* legacy URL: https://tools.wmflabs.org/openstack-browser/
* new URL: https://openstack-browser.toolforge.org
This domain change is something we have been working on for months previous to
this announcement. Part of our work has been to ensure we have a smooth
transition from the old domain (and URL scheme) to the new canonical one.
However, we acknowledge the ride might be bumpy for some folks, due to technical
challenges or cases we didn't consider when planning this migration. Please
reach out intermediately if you find any limitation or failure anywhere related
to this change. The wikitech page also contains a section with information for
common problems.
You can check now if your webservice needs any specific change by creating a
temporal redirection to the new canonical URL:
$ webservice --canonical --backend=kubernetes start [..]
$ webservice --canonical --backend=gridengine start [..]
The --canonical switch will create a temporal redirect that you can turn on/off.
Please use this to check how your webservice behaves with the new domain/URL
scheme. If you start the webservice without --canonical, the temporal redirect
will be removed.
We aim to introduce permanent redirects for the legacy URLs on 2020-06-15. We
expect to keep serving legacy URLs forever, by means of redirections to the new
URLs. More information on the redirections can also be found in the wikitech page.
The toolforge.org domain is finally here! <3
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
Hi,
we just enabled email ratelimiting in our MTA server [0] in Toolforge.
Please, report any problem or issue you may find related to this.
The current limit is 100 messages per hour per sender address. We may tune the
value as we observe the behavior of the system and the users.
regards.
[0] https://en.wikipedia.org/wiki/Message_transfer_agent
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation