On Tue, Aug 18, 2020 at 9:03 AM Bryan Davis <bd808(a)wikimedia.org> wrote:
>
> TL;DR:
> * HTTP -> HTTPS redirection is live (finally!)
> * Currently allowing a "POST loophole"
> * "POST loophole" will be closed on 2021-02-01
>
> Today we merged a small change [0] to the front proxy used by Cloud
> VPS projects [1]. This change brings automatic HTTP -> HTTPS
> redirection to the "domain proxy" service and a
> Strict-Transport-Security header with a 1 day duration.
>
> The current configuration is conservative. We will only redirect GET
> and HEAD requests to HTTPS to avoid triggering bugs in the handling of
> redirects during POST requests. This "POST loophole" is the same
> process that we followed when converting the production wiki farm and
> Toolforge to HTTPS.
>
> When we announced similar changes for Toolforge in 2019 [2] we forgot
> to set a timeline for closing the POST loophole. This time we are
> wiser! We will close the POST loophole and make all HTTP requests,
> regardless of the verb used, redirect to HTTPS on 2021-02-01. This 6
> month transition period should give us all a chance to find and update
> URLs to use https and to fix any dependent software that might break
> if a redirect was sent for a POST request.
>
> If you find issues in your projects resulting from this change, please
> do let us know. The tracking task for this change is T120486 [3]. We
> also provide support in the #wikimedia-cloud channel on Freenode and
> via the cloud(a)lists.wikimedia.org mailing list [4].
>
>
> [0]: https://gerrit.wikimedia.org/r/c/operations/puppet/+/620122/
> [1]: https://wikitech.wikimedia.org/wiki/Help:Using_a_web_proxy_to_reach_Cloud_V…
> [2]: https://phabricator.wikimedia.org/phame/post/view/132/migrating_tools.wmfla…
> [3]: https://phabricator.wikimedia.org/T120486
> [4]: https://lists.wikimedia.org/mailman/listinfo/cloud
TL;DR:
* "POST loophole" closed per prior announcement on 2020-08-18
* 366 day Strict-Transport-Security header sent with all HTTPS responses
I am very happy to announce that today we have closed the "POST
loophole" for our *.wmflabs.org & *.wmcloud.org proxy layer [5]. This
is a follow up to the announcement of partial TLS enforcement by the
Cloud VPS front proxies on 2020-08-18.
There is a possibility that closing the POST loophole will break some
clients accessing services running in Cloud VPS behind the front
proxies. Specifically, POST actions sent to HTTP (not HTTPS) URLs will
now return a 301 Moved Permanently response to the same URL with the
scheme changed to https. The HTTP specifications are ambiguous about
how this response should be handled which means that implementations
in various browsers and libraries may or may not re-POST the original
payload to the new URL. The best fix we can suggest for this is
updating links and forms to always use HTTPS URLs.
If you find issues in your projects resulting from this change, please
do let us know. The tracking task for this change is T120486 [6]. We
also provide support in the #wikimedia-cloud channel on Freenode and
via the cloud(a)lists.wikimedia.org mailing list [7].
[5]: https://gerrit.wikimedia.org/r/661140
[6]: https://phabricator.wikimedia.org/T120486
[7]: https://lists.wikimedia.org/mailman/listinfo/cloud
Bryan, on behalf of the Cloud VPS admin team
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
Hello,
we are planning to change how Cloud VPS instances and Toolforge tools contact
WMF-hosted wikis, in particular the source IP address for the network connection.
The new IP address that wikis will see is 185.15.56.1.
The change is scheduled to go live on 2021-02-08.
More detailed information in wikitech:
https://wikitech.wikimedia.org/wiki/News/CloudVPS_NAT_wikis
If you are a Cloud VPS user or Toolforge developer, check your tools after that
date to make sure they are properly running. If you detect a block, a rate-limit
or similar, please let us know.
If you are a WMF SRE or engineer involved with the wikis, be informed that this
address could generate a significant traffic volume, perhaps about 30%-40% total
wiki edits. We are trying to smooth the change as much as possible, so please
send your feedback if you think there is something we didn't account for yet.
Thanks, best regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
Hello!
The 2020 project opt-in process wrapped up at the end of the year, and
we've identified the following projects as abandoned:
- asyncwiki
- blog
- commons-corruption-checker
- fastcci
- finding-glams
- ign2commons
- lizenzhinweisgenerator
- lta-tracker
- meza
- ogvjs-integration
- puppet
- snuggle
- wikidata-federation
- wikidata-primary-sources-tool
- wikidata-realtime-dumps
- wikimania-scholarships
At the end of this month (2020-01-31) those projects will be deleted
along with all related data and VMs. If you know of anyone associated
with those projects who is not on this list, please bring this to their
attention. And, if you think any of this is in error, please notify me
immediately.
Thank you!
-Andrew + the WMCS team
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
Hi everyone.
We have some updates related to T260389 Redesign and rebuild the
wikireplicas service using a multi-instance architecture
<https://phabricator.wikimedia.org/T260389>.
The new cluster will be ready for testing on February 1st, and we need your
help.
Please subscribe and comment on T272523 Early testing of the multi-instance
architecture <https://phabricator.wikimedia.org/T272523> or reach out if
you would like to help test it by migrating your code. We want to closely
monitor it and tune it as we ramp up usage.
PAWS and Quarry will migrate to use the new cluster later in February, we
will publish more information as soon as possible.
The old cluster will remain available in parallel during the migration.
Thank you for your help,
--
Joaquin Oltra Hernandez
Developer Advocate - Wikimedia Foundation
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
This Thursday we will be upgrading the cloud-vps OpenStack install to
version 'Stein'. During the upgrade window (probably about an hour),
Horizon will be disabled.
Existing tools and VMs should be largely unaffected, although there may
be a brief network interruption when the routing services restart.
The upgrade is scheduled for 15:00 UTC, which is 7AM in California.
-Andrew + the WMCS team
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
Hi Everyone,
We’re happy to announce the January 2021 edition of the Technical Community
Newsletter
<https://www.mediawiki.org/wiki/Technical_Community_Newsletter/2021/January>
is now available. The newsletter is compiled by the Wikimedia Developer
Advocacy Team. It aims to share highlights, news, and information of
interest from and about the Wikimedia technical community.
Check it out, and learn about what technical contributors have been up to
this past quarter, upcoming conferences & calls for papers, and how to get
involved.
The Wikimedia Technical Community is large and diverse, and we know we
can't capture everything perfectly. We welcome your ideas for future
newsletters. Let us know what you would like to see or highlights you would
like us to include.
Subscribe to the Technical Community Newsletter
<https://www.mediawiki.org/wiki/Newsletter:Technical_Community_Newsletter>,
if you'd like to keep up with essential updates and information
Kindly,
Sarah R. Rodlund
Senior Technical Writer, Developer Advocacy
<https://www.mediawiki.org/wiki/Developer_Advocacy>
srodlund(a)wikimedia.org
My toolforge service (https://author-disambiguator.toolforge.org/) keeps
becoming unavailable with hangs/502 Bad Gateway or other server errors a
few minutes after I restart it, and I can't see what could be causing this.
These errors don't show up in the error log, and the 502 responses don't
show up in the access log (which has had very little traffic anyway - one
request per minute at most usually?) I can connect to the kubernetes pod
with kubectl and everything looks normal,there's only a few processes
listed in /proc, etc. (though it would be nice to have some other
monitoring tools like ps and netstat installed by default?) But I can't get
a response via the web after the first few minutes.
The problem seems to have started mid-day yesterday - see the monitor data
here:
https://grafana-labs.wikimedia.org/d/toolforge-k8s-namespace-resources/kube…
with the surge in 4xx and 5xx status codes on 1/3 (by the way, I don't see
the surge in 4xx status codes in access.log recently either - there are 2
from this morning and none yesterday, nothing like the multiple per second
indicated in that grafana chart!)
Any ideas what's going on? This looks like some sort of upstream issue with
nginx maybe?
I am seeing a "You have run out of local ports" error in the error logs
from earlier today (but it hasn't repeated recently) which is maybe a clue?
I don't think that could possibly be from anything my service is doing
though!
Help would be greatly appreciated, thanks!
Arthur Smith
We will be failing over the Toolforge and Project NFS in 10 minutes to move the main interface to 10Gb Ethernet. The previous work should make this fairly non-disruptive, but that was believed in the past as well.
Brooke Storm
Cloud Service Team
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
Is there a problem with SSH on {dev,login}.tools.wmflabs.org <http://tools.wmflabs.org/>? I can ping both of those hosts, but SSH is not responding for me.
Question: Is it permitted to use WMCS (be it Cloud VPS or Toolforge) resources for non-Wikimedia, Mediawiki-run website purposes? (Let's assume sites like FANDOM(Wikia), Miraheze, or https://librewiki.net )
Background: I was discussing some non-wikimedia (but mediawiki-run site) stuff at some chatroom. Someone there thought they could use Wikimedia Cloud resources for non-WMF websites (I don't know if it relates to MW development) purposes. I am not sure if this is allowed (my interpretation is that it is not allowed), so I want some clarification on that.
나의 iPhone에서 보냄