Cloud-announce April 2020

cloud-announce@lists.wikimedia.org

4 participants
5 discussions

Toolforge: new domain toolforge.org
by Arturo Borrero Gonzalez 07 Jul '20

07 Jul '20

Hi! We are happy to announce the new domain 'toolforge.org' is now ready to be adopted by our Toolforge community. There is a lot of information related to this change in a wikitech page we have for this: https://wikitech.wikimedia.org/wiki/News/Toolforge.org The most important change you will see happening is a new domain/scheme for Toolforge-hosted webservices: * from https://tools.wmflabs.org/<toolname>/ * to https://<toolname>.toolforge.org/ A live example of this change can be found in our internal openstack-browser webservice tool: * legacy URL: https://tools.wmflabs.org/openstack-browser/ * new URL: https://openstack-browser.toolforge.org This domain change is something we have been working on for months previous to this announcement. Part of our work has been to ensure we have a smooth transition from the old domain (and URL scheme) to the new canonical one. However, we acknowledge the ride might be bumpy for some folks, due to technical challenges or cases we didn't consider when planning this migration. Please reach out intermediately if you find any limitation or failure anywhere related to this change. The wikitech page also contains a section with information for common problems. You can check now if your webservice needs any specific change by creating a temporal redirection to the new canonical URL: $ webservice --canonical --backend=kubernetes start [..] $ webservice --canonical --backend=gridengine start [..] The --canonical switch will create a temporal redirect that you can turn on/off. Please use this to check how your webservice behaves with the new domain/URL scheme. If you start the webservice without --canonical, the temporal redirect will be removed. We aim to introduce permanent redirects for the legacy URLs on 2020-06-15. We expect to keep serving legacy URLs forever, by means of redirections to the new URLs. More information on the redirections can also be found in the wikitech page. The toolforge.org domain is finally here! <3 -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

2 6

Disk failure on cloudvirt1004 OpenStack host
by Bryan Davis 21 Apr '20

21 Apr '20

cloudvirt1004 is one of our oldest generation of hypervisor servers. The hypervisor servers are the machines which actually run the virtual machine instances for Cloud VPS projects. This physical host is experiencing an active hard disk and/or RAID controller failure. The Cloud Services team is actively attempting to fix the server and evacuate all instances running on it to other hypervisors. See <https://phabricator.wikimedia.org/T250869> for more information and progress updates. The following projects and instances are affected: * cloudvirt-canary ** canary1004-01.cloudvirt-canary.eqiad.wmflabs * commonsarchive ** commonsarchive-mwtest.commonsarchive.eqiad.wmflabs * deployment-prep ** deployment-echostore01.deployment-prep.eqiad.wmflabs ** deployment-schema-2.deployment-prep.eqiad.wmflabs * incubator ** incubator-mw.incubator.eqiad.wmflabs * machine-vision ** visionoid.machine-vision.eqiad.wmflabs * ogvjs-integration ** media-streaming.ogvjs-integration.eqiad.wmflabs * services ** Esther-outreachy-intern.services.eqiad.wmflabs * shiny-r ** discovery-testing-02.shiny-r.eqiad.wmflabs * tools ** tools-k8s-worker-38.tools.eqiad.wmflabs ** tools-k8s-worker-52.tools.eqiad.wmflabs ** tools-sgeexec-0901.tools.eqiad.wmflabs ** tools-sgewebgrid-lighttpd-0918.tools.eqiad.wmflabs ** tools-sgewebgrid-lighttpd-0919.tools.eqiad.wmflabs * toolsbeta ** toolsbeta-sgewebgrid-generic-0901.toolsbeta.eqiad.wmflabs * wikidata-autodesc ** wikidata-autodesc.wikidata-autodesc.eqiad.wmflabs * wikilink ** wikilink-prod.wikilink.eqiad.wmflabs Bryan, on behalf of the Cloud VPS admins and Cloud Services team -- Bryan Davis Technical Engagement Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808

1 0

cloud-vps maintenance tomorrow, 2019-04-16
by Andrew Bogott 16 Apr '20

16 Apr '20

We'll be upgrading the cloud services OpenStack install tomorrow, beginning at 15:00 UTC. There should be little to no interruption to VMs or Toolforge, but Horizon logins will be disabled for part of the window. Sorry for the short notice! - Andrew + the WMCS team

1 2

Changes to CloudVPS web proxy (XFF) on 2020-04-15
by Arturo Borrero Gonzalez 15 Apr '20

15 Apr '20

Hi there! If you use a CloudVPS web proxy, this email is for you. Toolforge developers/users can ignore this email. We are introducing a change to eliminate the 'X-Forwarded-For' HTTP header that the CloudVPS web proxy adds when forwarding the HTTP request to your instance. This header contains the original IP address of the internet client that sent the request. This is private information that we would like to reduce in our environment [0]. You use the web proxy if you have a public web endpoint hosted in CloudVPS under the wmflabs.org domain. These are generally configured using Horizon in the DNS > Web Proxies section. Examples of web proxy names: * accounts.wmflabs.org * glampipe.wmflabs.org * incubator.wmflabs.org Full list can be seen in the Openstack Browser tool [1]. We are ready to introduce this change [2], but wanted to give some heads up for projects that do require this information for whatever reason. We would like to hear from you in the next couple of weeks. Please contact us in the phabricator task [0] and include some rationale why you need the XFF header. This is the timeline this change will follow: * 2020-04-01: this email, start collecting list of things that require XFF * 2020-04-07: start evaluating list of things that require XFF * 2020-04-15: introduce the change, with proper case whitelisting When the change is introduced, in two weeks from now, proxy backends that were not whitelisted will stop receiving the XFF header. Please reach out for any questions or comments. regards. [0] https://phabricator.wikimedia.org/T135046 [1] https://openstack-browser.toolforge.org/project/project-proxy [2] https://gerrit.wikimedia.org/r/c/operations/puppet/+/583098 -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 1

CloudVPS network change (routing source IP) on 2020-04-13
by Arturo Borrero Gonzalez 09 Apr '20

09 Apr '20

Hi there! In a few days from now (2020-04-13), the CloudVPS network will see a change happening that will likely go unnoticed, but it is important enough to share it with you beforehand. We will be changing the IPv4 address that we use as the main source NAT for egress connections (initiated in the VM instances). This change won't affect VM instances using floating IPs. Old IP address: 185.15.56.1 New IP address: 208.80.155.92 If you know of anywhere (a firewall, ACL or any other mechanism) that had this address hardcoded, you will need to update it. See this wikitech page for more details: https://wikitech.wikimedia.org/wiki/News/CloudVPS_NAT_change Please reach out if you have any doubts, questions, or any other issue. regards. -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 1

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-announce April 2020