On Tue, Aug 18, 2020 at 9:03 AM Bryan Davis <bd808(a)wikimedia.org> wrote:
>
> TL;DR:
> * HTTP -> HTTPS redirection is live (finally!)
> * Currently allowing a "POST loophole"
> * "POST loophole" will be closed on 2021-02-01
>
> Today we merged a small change [0] to the front proxy used by Cloud
> VPS projects [1]. This change brings automatic HTTP -> HTTPS
> redirection to the "domain proxy" service and a
> Strict-Transport-Security header with a 1 day duration.
>
> The current configuration is conservative. We will only redirect GET
> and HEAD requests to HTTPS to avoid triggering bugs in the handling of
> redirects during POST requests. This "POST loophole" is the same
> process that we followed when converting the production wiki farm and
> Toolforge to HTTPS.
>
> When we announced similar changes for Toolforge in 2019 [2] we forgot
> to set a timeline for closing the POST loophole. This time we are
> wiser! We will close the POST loophole and make all HTTP requests,
> regardless of the verb used, redirect to HTTPS on 2021-02-01. This 6
> month transition period should give us all a chance to find and update
> URLs to use https and to fix any dependent software that might break
> if a redirect was sent for a POST request.
>
> If you find issues in your projects resulting from this change, please
> do let us know. The tracking task for this change is T120486 [3]. We
> also provide support in the #wikimedia-cloud channel on Freenode and
> via the cloud(a)lists.wikimedia.org mailing list [4].
>
>
> [0]: https://gerrit.wikimedia.org/r/c/operations/puppet/+/620122/
> [1]: https://wikitech.wikimedia.org/wiki/Help:Using_a_web_proxy_to_reach_Cloud_V…
> [2]: https://phabricator.wikimedia.org/phame/post/view/132/migrating_tools.wmfla…
> [3]: https://phabricator.wikimedia.org/T120486
> [4]: https://lists.wikimedia.org/mailman/listinfo/cloud
TL;DR:
* "POST loophole" closed per prior announcement on 2020-08-18
* 366 day Strict-Transport-Security header sent with all HTTPS responses
I am very happy to announce that today we have closed the "POST
loophole" for our *.wmflabs.org & *.wmcloud.org proxy layer [5]. This
is a follow up to the announcement of partial TLS enforcement by the
Cloud VPS front proxies on 2020-08-18.
There is a possibility that closing the POST loophole will break some
clients accessing services running in Cloud VPS behind the front
proxies. Specifically, POST actions sent to HTTP (not HTTPS) URLs will
now return a 301 Moved Permanently response to the same URL with the
scheme changed to https. The HTTP specifications are ambiguous about
how this response should be handled which means that implementations
in various browsers and libraries may or may not re-POST the original
payload to the new URL. The best fix we can suggest for this is
updating links and forms to always use HTTPS URLs.
If you find issues in your projects resulting from this change, please
do let us know. The tracking task for this change is T120486 [6]. We
also provide support in the #wikimedia-cloud channel on Freenode and
via the cloud(a)lists.wikimedia.org mailing list [7].
[5]: https://gerrit.wikimedia.org/r/661140
[6]: https://phabricator.wikimedia.org/T120486
[7]: https://lists.wikimedia.org/mailman/listinfo/cloud
Bryan, on behalf of the Cloud VPS admin team
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
Hello,
we are planning to change how Cloud VPS instances and Toolforge tools contact
WMF-hosted wikis, in particular the source IP address for the network connection.
The new IP address that wikis will see is 185.15.56.1.
The change is scheduled to go live on 2021-02-08.
More detailed information in wikitech:
https://wikitech.wikimedia.org/wiki/News/CloudVPS_NAT_wikis
If you are a Cloud VPS user or Toolforge developer, check your tools after that
date to make sure they are properly running. If you detect a block, a rate-limit
or similar, please let us know.
If you are a WMF SRE or engineer involved with the wikis, be informed that this
address could generate a significant traffic volume, perhaps about 30%-40% total
wiki edits. We are trying to smooth the change as much as possible, so please
send your feedback if you think there is something we didn't account for yet.
Thanks, best regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
Hello!
The 2020 project opt-in process wrapped up at the end of the year, and
we've identified the following projects as abandoned:
- asyncwiki
- blog
- commons-corruption-checker
- fastcci
- finding-glams
- ign2commons
- lizenzhinweisgenerator
- lta-tracker
- meza
- ogvjs-integration
- puppet
- snuggle
- wikidata-federation
- wikidata-primary-sources-tool
- wikidata-realtime-dumps
- wikimania-scholarships
At the end of this month (2020-01-31) those projects will be deleted
along with all related data and VMs. If you know of anyone associated
with those projects who is not on this list, please bring this to their
attention. And, if you think any of this is in error, please notify me
immediately.
Thank you!
-Andrew + the WMCS team
Hi everyone.
We have some updates related to T260389 Redesign and rebuild the
wikireplicas service using a multi-instance architecture
<https://phabricator.wikimedia.org/T260389>.
The new cluster will be ready for testing on February 1st, and we need your
help.
Please subscribe and comment on T272523 Early testing of the multi-instance
architecture <https://phabricator.wikimedia.org/T272523> or reach out if
you would like to help test it by migrating your code. We want to closely
monitor it and tune it as we ramp up usage.
PAWS and Quarry will migrate to use the new cluster later in February, we
will publish more information as soon as possible.
The old cluster will remain available in parallel during the migration.
Thank you for your help,
--
Joaquin Oltra Hernandez
Developer Advocate - Wikimedia Foundation
This Thursday we will be upgrading the cloud-vps OpenStack install to
version 'Stein'. During the upgrade window (probably about an hour),
Horizon will be disabled.
Existing tools and VMs should be largely unaffected, although there may
be a brief network interruption when the routing services restart.
The upgrade is scheduled for 15:00 UTC, which is 7AM in California.
-Andrew + the WMCS team
We will be failing over the Toolforge and Project NFS in 10 minutes to move the main interface to 10Gb Ethernet. The previous work should make this fairly non-disruptive, but that was believed in the past as well.
Brooke Storm
Cloud Service Team
In yet another effort to restore replication and preserve the redundancy of the data in ToolsDB (user writable database in Toolforge), we need to take the database (tools.db.svc.eqiad.wmflabs) completely offline at 1700 UTC on 16 Dec. Apps that depend on the ToolsDB service will fail during the outage (which will take at least an hour, and we aren’t entirely sure exactly how long—expect multiple hours). This will be much faster than the last outage because we are doing a straight copy of the binary database files between the servers. Details of this mess and efforts to restore the replication service can be found at https://phabricator.wikimedia.org/T266587 <https://phabricator.wikimedia.org/T266587>
If we succeed in producing a viable copy of the database on another system, we will also perform an upgrade on the hypervisor it is on before closing the maintenance period. That should be an additional hour or so.
We appreciate your patience with this process. It is very important that we establish a second copy of this database, especially in light of recent crashes (https://phabricator.wikimedia.org/T253738 <https://phabricator.wikimedia.org/T253738>).
Brooke Storm
Staff SRE
Wikimedia Cloud Services
bstorm(a)wikimedia.org
IRC: bstorm_
Hi there!
Today 2020-12-10 @ 15:30 UTC we will perform an upgrade of the Toolforge
kubernetes cluster [0].
We don't expect any major disruption of the service, but we detected in past
upgrades that some components might be restarted, causing brief interruptions of
network flows.
Given the amount of worker nodes we have, more than 50, the operation will take
us at least a couple of hours.
Tools maintainers: you don't have to do anything during this operation, but if
you detect anything weird please contact us either in the phabricator task [0],
in the IRC channel #wikimedia-cloud or in the cloud(a)lists.wikimedia.org [1]
mailing list.
regards.
[0] https://phabricator.wikimedia.org/T263284
[1] https://lists.wikimedia.org/mailman/listinfo/cloud
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
Hi there,
we are about to upgrade the kubernetes version that runs PAWS, from 1.6 to 1.17.
We don't expect any interruptions major on the service, perhaps only some
hiccups when pods are restarted/rescheduled.
More information is available in this phabricator ticket:
https://phabricator.wikimedia.org/T268669
The operation may take something between 30 minutes and 1 hours, and we are
starting soon after I finish sending this email.
Please, ping us if you see anything wrong.
regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
Every year or so the Cloud Services team tries to identify and clean up
unused projects and VMs. We do this via an opt-in process: anyone can
mark a project as 'in use,' and that project will be preserved for
another year.
I've created a wiki page the lists all existing projects, here:
https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge
If you are a VPS user, please visit that page and mark any projects that
you use as {{Used}}. Note that it's not necessary for you to be a
project admin to mark something -- if you know that you're currently
using a resource and want to keep using it, go ahead and mark it
accordingly. If you /are/ a project admin, please take a moment to mark
which VMs are or aren't used in your projects.
When December arrives, I will shut down and begin the process of
reclaiming resources from unused projects.
If you think you use a VPS project but aren't sure which, I encourage
you to poke around on https://tools.wmflabs.org/openstack-browser/ to
see what looks familiar. Worst case, just email
cloud(a)lists.wikimedia.org with a description of your use case and we'll
sort it out there.
Exclusive toolforge users are free to ignore this task.
Thank you!
-Andrew and WMCS team