For security reasons we will be updating and rebooting many (possibly
all) cloud-vps VMs today, over the weekend, and on Monday.
The first step of this will be a reboot of all bastions (including the
toolforge login hosts) in about an hour, at 18:30 UTC. That means that
all ssh sessions to cloud-vps hosts will be interrupted as proxy
sessions are terminated.
After the tools bastion reboots, users of toolforge should not be
further affected by these reboots. All other cloud-vps users can expect
at least one more surprise reboot over the next few days. If you need
that reboot to be scheduled at a particular time please contact me
immediately.
Sorry for the inconvenience!
-Andrew + the WMCS team
Hello,
today 2021-02-23 in about ~30 minutes (16:00 UTC) we will change how virtual
machine instances running in Cloud VPS contact NFS dump servers [0].
There is no action required on your side.
We anticipate little to no impact as a result of the network changes. But in
case you notice something is not properly working with dumps NFS in Cloud VPS
(or Toolforge) please contact us [1] as soon as possible. The relevant
phabricator ticket [2] is T272397.
regards.
[0] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Dumps
[1]
https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_Introduction#Commun…
[2] https://phabricator.wikimedia.org/T272397
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
On Tuesday we will be upgrading the cloud-vps OpenStack install to
version 'Train'. During the upgrade window (probably about an hour),
Horizon will be disabled.
Existing tools and VMs should be largely unaffected; there will likely
be a brief network interruption when the network software is restarted.
The upgrade is scheduled for 15:00 UTC, which is 7AM in California.
-Andrew + the WMCS team
Wikimedia Cloud Services now supports attachable block storage via the
OpenStack Cinder project. Attachable block storage is a flexible storage
option that allows you to create volumes local to your project but not
coupled to a particular VM; they can be moved between different
instances and persist after their associated volume is deleted. Project
admins can access this feature via the 'Volumes' tab in Horizon.
I encourage all of you to start using Cinder storage for your new
databases and large data sets. Over the next few months we'll be working
to move various use cases onto Cinder volumes and off of NFS or LVM;
soon I hope to deprecate large-storage flavor types entirely and support
all new non-root file storage with Cinder.
The default storage quota is quite small, but we plan to be generous
with quota increases. To request additional storage, open a phabricator
ticket here:
https://phabricator.wikimedia.org/project/view/2880/
For more details about this feature, I've written a blog post, here:
https://techblog.wikimedia.org/2021/02/05/cinder-on-cloud-vps/
And, technical documentation can be found here:
https://wikitech.wikimedia.org/wiki/Help:Adding_Disk_Space_to_Cloud_VPS_ins…
Like any new feature, our implementation almost certainly includes bugs
and missteps. Please provide feedback or feature requests via
phabricator or on the cloud mailing list.
-Andrew + the WMCS Team
On Tue, Aug 18, 2020 at 9:03 AM Bryan Davis <bd808(a)wikimedia.org> wrote:
>
> TL;DR:
> * HTTP -> HTTPS redirection is live (finally!)
> * Currently allowing a "POST loophole"
> * "POST loophole" will be closed on 2021-02-01
>
> Today we merged a small change [0] to the front proxy used by Cloud
> VPS projects [1]. This change brings automatic HTTP -> HTTPS
> redirection to the "domain proxy" service and a
> Strict-Transport-Security header with a 1 day duration.
>
> The current configuration is conservative. We will only redirect GET
> and HEAD requests to HTTPS to avoid triggering bugs in the handling of
> redirects during POST requests. This "POST loophole" is the same
> process that we followed when converting the production wiki farm and
> Toolforge to HTTPS.
>
> When we announced similar changes for Toolforge in 2019 [2] we forgot
> to set a timeline for closing the POST loophole. This time we are
> wiser! We will close the POST loophole and make all HTTP requests,
> regardless of the verb used, redirect to HTTPS on 2021-02-01. This 6
> month transition period should give us all a chance to find and update
> URLs to use https and to fix any dependent software that might break
> if a redirect was sent for a POST request.
>
> If you find issues in your projects resulting from this change, please
> do let us know. The tracking task for this change is T120486 [3]. We
> also provide support in the #wikimedia-cloud channel on Freenode and
> via the cloud(a)lists.wikimedia.org mailing list [4].
>
>
> [0]: https://gerrit.wikimedia.org/r/c/operations/puppet/+/620122/
> [1]: https://wikitech.wikimedia.org/wiki/Help:Using_a_web_proxy_to_reach_Cloud_V…
> [2]: https://phabricator.wikimedia.org/phame/post/view/132/migrating_tools.wmfla…
> [3]: https://phabricator.wikimedia.org/T120486
> [4]: https://lists.wikimedia.org/mailman/listinfo/cloud
TL;DR:
* "POST loophole" closed per prior announcement on 2020-08-18
* 366 day Strict-Transport-Security header sent with all HTTPS responses
I am very happy to announce that today we have closed the "POST
loophole" for our *.wmflabs.org & *.wmcloud.org proxy layer [5]. This
is a follow up to the announcement of partial TLS enforcement by the
Cloud VPS front proxies on 2020-08-18.
There is a possibility that closing the POST loophole will break some
clients accessing services running in Cloud VPS behind the front
proxies. Specifically, POST actions sent to HTTP (not HTTPS) URLs will
now return a 301 Moved Permanently response to the same URL with the
scheme changed to https. The HTTP specifications are ambiguous about
how this response should be handled which means that implementations
in various browsers and libraries may or may not re-POST the original
payload to the new URL. The best fix we can suggest for this is
updating links and forms to always use HTTPS URLs.
If you find issues in your projects resulting from this change, please
do let us know. The tracking task for this change is T120486 [6]. We
also provide support in the #wikimedia-cloud channel on Freenode and
via the cloud(a)lists.wikimedia.org mailing list [7].
[5]: https://gerrit.wikimedia.org/r/661140
[6]: https://phabricator.wikimedia.org/T120486
[7]: https://lists.wikimedia.org/mailman/listinfo/cloud
Bryan, on behalf of the Cloud VPS admin team
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
Hello,
we are planning to change how Cloud VPS instances and Toolforge tools contact
WMF-hosted wikis, in particular the source IP address for the network connection.
The new IP address that wikis will see is 185.15.56.1.
The change is scheduled to go live on 2021-02-08.
More detailed information in wikitech:
https://wikitech.wikimedia.org/wiki/News/CloudVPS_NAT_wikis
If you are a Cloud VPS user or Toolforge developer, check your tools after that
date to make sure they are properly running. If you detect a block, a rate-limit
or similar, please let us know.
If you are a WMF SRE or engineer involved with the wikis, be informed that this
address could generate a significant traffic volume, perhaps about 30%-40% total
wiki edits. We are trying to smooth the change as much as possible, so please
send your feedback if you think there is something we didn't account for yet.
Thanks, best regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation