Cloud-announce February 2021

cloud-announce@lists.wikimedia.org

3 participants
6 discussions

ssh session interruptions today
by Andrew Bogott 27 Feb '21

27 Feb '21

For security reasons we will be updating and rebooting many (possibly all) cloud-vps VMs today, over the weekend, and on Monday. The first step of this will be a reboot of all bastions (including the toolforge login hosts) in about an hour, at 18:30 UTC. That means that all ssh sessions to cloud-vps hosts will be interrupted as proxy sessions are terminated. After the tools bastion reboots, users of toolforge should not be further affected by these reboots. All other cloud-vps users can expect at least one more surprise reboot over the next few days. If you need that reboot to be scheduled at a particular time please contact me immediately. Sorry for the inconvenience! -Andrew + the WMCS team

1 0

Network change for VMs contacting NFS dumps
by Arturo Borrero Gonzalez 23 Feb '21

23 Feb '21

Hello, today 2021-02-23 in about ~30 minutes (16:00 UTC) we will change how virtual machine instances running in Cloud VPS contact NFS dump servers [0]. There is no action required on your side. We anticipate little to no impact as a result of the network changes. But in case you notice something is not properly working with dumps NFS in Cloud VPS (or Toolforge) please contact us [1] as soon as possible. The relevant phabricator ticket [2] is T272397. regards. [0] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Dumps [1] https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_Introduction#Commun… [2] https://phabricator.wikimedia.org/T272397 -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 0

cloud-vps maintenance scheduled for Tuesday, 15:00 UTC
by Andrew Bogott 10 Feb '21

10 Feb '21

On Tuesday we will be upgrading the cloud-vps OpenStack install to version 'Train'. During the upgrade window (probably about an hour), Horizon will be disabled. Existing tools and VMs should be largely unaffected; there will likely be a brief network interruption when the network software is restarted. The upgrade is scheduled for 15:00 UTC, which is 7AM in California. -Andrew + the WMCS team

1 2

new Cloud VPS feature: attachable block storage
by Andrew Bogott 05 Feb '21

05 Feb '21

Wikimedia Cloud Services now supports attachable block storage via the OpenStack Cinder project. Attachable block storage is a flexible storage option that allows you to create volumes local to your project but not coupled to a particular VM; they can be moved between different instances and persist after their associated volume is deleted. Project admins can access this feature via the 'Volumes' tab in Horizon. I encourage all of you to start using Cinder storage for your new databases and large data sets. Over the next few months we'll be working to move various use cases onto Cinder volumes and off of NFS or LVM; soon I hope to deprecate large-storage flavor types entirely and support all new non-root file storage with Cinder. The default storage quota is quite small, but we plan to be generous with quota increases. To request additional storage, open a phabricator ticket here: https://phabricator.wikimedia.org/project/view/2880/ For more details about this feature, I've written a blog post, here: https://techblog.wikimedia.org/2021/02/05/cinder-on-cloud-vps/ And, technical documentation can be found here: https://wikitech.wikimedia.org/wiki/Help:Adding_Disk_Space_to_Cloud_VPS_ins… Like any new feature, our implementation almost certainly includes bugs and missteps. Please provide feedback or feature requests via phabricator or on the cloud mailing list. -Andrew + the WMCS Team

1 0

[Cloud VPS] TLS encryption fully enforced for *.wmflabs.org & *.wmcloud.org
by Bryan Davis 03 Feb '21

03 Feb '21

On Tue, Aug 18, 2020 at 9:03 AM Bryan Davis <bd808(a)wikimedia.org> wrote: > > TL;DR: > * HTTP -> HTTPS redirection is live (finally!) > * Currently allowing a "POST loophole" > * "POST loophole" will be closed on 2021-02-01 > > Today we merged a small change [0] to the front proxy used by Cloud > VPS projects [1]. This change brings automatic HTTP -> HTTPS > redirection to the "domain proxy" service and a > Strict-Transport-Security header with a 1 day duration. > > The current configuration is conservative. We will only redirect GET > and HEAD requests to HTTPS to avoid triggering bugs in the handling of > redirects during POST requests. This "POST loophole" is the same > process that we followed when converting the production wiki farm and > Toolforge to HTTPS. > > When we announced similar changes for Toolforge in 2019 [2] we forgot > to set a timeline for closing the POST loophole. This time we are > wiser! We will close the POST loophole and make all HTTP requests, > regardless of the verb used, redirect to HTTPS on 2021-02-01. This 6 > month transition period should give us all a chance to find and update > URLs to use https and to fix any dependent software that might break > if a redirect was sent for a POST request. > > If you find issues in your projects resulting from this change, please > do let us know. The tracking task for this change is T120486 [3]. We > also provide support in the #wikimedia-cloud channel on Freenode and > via the cloud(a)lists.wikimedia.org mailing list [4]. > > > [0]: https://gerrit.wikimedia.org/r/c/operations/puppet/+/620122/ > [1]: https://wikitech.wikimedia.org/wiki/Help:Using_a_web_proxy_to_reach_Cloud_V… > [2]: https://phabricator.wikimedia.org/phame/post/view/132/migrating_tools.wmfla… > [3]: https://phabricator.wikimedia.org/T120486 > [4]: https://lists.wikimedia.org/mailman/listinfo/cloud TL;DR: * "POST loophole" closed per prior announcement on 2020-08-18 * 366 day Strict-Transport-Security header sent with all HTTPS responses I am very happy to announce that today we have closed the "POST loophole" for our *.wmflabs.org & *.wmcloud.org proxy layer [5]. This is a follow up to the announcement of partial TLS enforcement by the Cloud VPS front proxies on 2020-08-18. There is a possibility that closing the POST loophole will break some clients accessing services running in Cloud VPS behind the front proxies. Specifically, POST actions sent to HTTP (not HTTPS) URLs will now return a 301 Moved Permanently response to the same URL with the scheme changed to https. The HTTP specifications are ambiguous about how this response should be handled which means that implementations in various browsers and libraries may or may not re-POST the original payload to the new URL. The best fix we can suggest for this is updating links and forms to always use HTTPS URLs. If you find issues in your projects resulting from this change, please do let us know. The tracking task for this change is T120486 [6]. We also provide support in the #wikimedia-cloud channel on Freenode and via the cloud(a)lists.wikimedia.org mailing list [7]. [5]: https://gerrit.wikimedia.org/r/661140 [6]: https://phabricator.wikimedia.org/T120486 [7]: https://lists.wikimedia.org/mailman/listinfo/cloud Bryan, on behalf of the Cloud VPS admin team -- Bryan Davis Technical Engagement Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808

1 0

Change to how Cloud VPS and Toolforge contact Wikis
by Arturo Borrero Gonzalez 02 Feb '21

02 Feb '21

Hello, we are planning to change how Cloud VPS instances and Toolforge tools contact WMF-hosted wikis, in particular the source IP address for the network connection. The new IP address that wikis will see is 185.15.56.1. The change is scheduled to go live on 2021-02-08. More detailed information in wikitech: https://wikitech.wikimedia.org/wiki/News/CloudVPS_NAT_wikis If you are a Cloud VPS user or Toolforge developer, check your tools after that date to make sure they are properly running. If you detect a block, a rate-limit or similar, please let us know. If you are a WMF SRE or engineer involved with the wikis, be informed that this address could generate a significant traffic volume, perhaps about 30%-40% total wiki edits. We are trying to smooth the change as much as possible, so please send your feedback if you think there is something we didn't account for yet. Thanks, best regards. -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 1

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-announce February 2021