- Cloud-announce - lists.wikimedia.org

Changes to CloudVPS web proxy (XFF) on 2020-04-15
by Arturo Borrero Gonzalez 15 Apr '20

15 Apr '20

Hi there! If you use a CloudVPS web proxy, this email is for you. Toolforge developers/users can ignore this email. We are introducing a change to eliminate the 'X-Forwarded-For' HTTP header that the CloudVPS web proxy adds when forwarding the HTTP request to your instance. This header contains the original IP address of the internet client that sent the request. This is private information that we would like to reduce in our environment [0]. You use the web proxy if you have a public web endpoint hosted in CloudVPS under the wmflabs.org domain. These are generally configured using Horizon in the DNS > Web Proxies section. Examples of web proxy names: * accounts.wmflabs.org * glampipe.wmflabs.org * incubator.wmflabs.org Full list can be seen in the Openstack Browser tool [1]. We are ready to introduce this change [2], but wanted to give some heads up for projects that do require this information for whatever reason. We would like to hear from you in the next couple of weeks. Please contact us in the phabricator task [0] and include some rationale why you need the XFF header. This is the timeline this change will follow: * 2020-04-01: this email, start collecting list of things that require XFF * 2020-04-07: start evaluating list of things that require XFF * 2020-04-15: introduce the change, with proper case whitelisting When the change is introduced, in two weeks from now, proxy backends that were not whitelisted will stop receiving the XFF header. Please reach out for any questions or comments. regards. [0] https://phabricator.wikimedia.org/T135046 [1] https://openstack-browser.toolforge.org/project/project-proxy [2] https://gerrit.wikimedia.org/r/c/operations/puppet/+/583098 -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 1

CloudVPS network change (routing source IP) on 2020-04-13
by Arturo Borrero Gonzalez 09 Apr '20

09 Apr '20

Hi there! In a few days from now (2020-04-13), the CloudVPS network will see a change happening that will likely go unnoticed, but it is important enough to share it with you beforehand. We will be changing the IPv4 address that we use as the main source NAT for egress connections (initiated in the VM instances). This change won't affect VM instances using floating IPs. Old IP address: 185.15.56.1 New IP address: 208.80.155.92 If you know of anywhere (a firewall, ACL or any other mechanism) that had this address hardcoded, you will need to update it. See this wikitech page for more details: https://wikitech.wikimedia.org/wiki/News/CloudVPS_NAT_change Please reach out if you have any doubts, questions, or any other issue. regards. -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 1

cloud-vps maintenance Thursday, 2019-03-19
by Andrew Bogott 26 Mar '20

26 Mar '20

Next Thursday we'll be upgrading to OpenStack version 'Queens' starting at 15:00 UTC. The upgrade will take a couple of hours. During the upgrade process, Horizon (and associated OpenStack APIs) will be disabled. There may also be brief network interruptions during the upgrade. Toolforge and existing VMs should be largely unaffected if all goes well -- there may be some service interruption if we encounter networking issues. - Andrew + the WMCS team

1 3

[Blog post] Tale of redesigning the huge Wikidata wb_terms table
by Léa Lacroix 23 Mar '20

23 Mar '20

Hello all, For almost a year, the Wikidata development team has been working on the task of redesigning and migrating the wb_terms table, which had become too big and unsustainable over the years. You can read the tale of our journey on this blog post: Come to Terms with Changes <https://phabricator.wikimedia.org/phame/post/view/195/coming_to_terms_with_…> If you’re a tool maintainer and your tool queries directly the Labs database replicas, you can read more details <https://lists.wikimedia.org/pipermail/wikidata/2020-March/013901.html> about the next steps and how to update your code. Congratulations to all the developers involved in this big project! Cheers, -- Léa Lacroix Project Manager Community Communication for Wikidata Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

1 0

[Toolforge] 2020 Kubernetes cluster migration complete
by Bryan Davis 06 Mar '20

06 Mar '20

On 2020-03-03, Brooke completed the automatic migration phase of the 2020 Kubernetes migration by moving the last workloads from the legacy Kubernetes cluster to the 2020 Kubernetes cluster [0]. All Toolforge tools using `webservice --backend=kubernetes ...` and/or manually maintained Kubernetes objects are now running on the 2020 Kubernetes cluster. The Toolforge admin team is in the process of tearing down the legacy cluster and cleaning up various documentation and tooling related to it [1]. This project involved a lot of hard work that most of the Toolforge community did not see. Brooke and Arturo started planning things over a year ago [2] to ensure that the Toolforge admin team would be able to complete this migration with a minimum amount of disruption to tools and their maintainers. Along the journey they researched Kubernetes best practices and recommendations, read and re-read numerous tutorial and how-to docs, and designed a completely new process to automate the deployment of Kubernetes in Toolforge. They also sought and received help from other Toolforge admins, Wikimedia Foundation staff, and technical volunteers. This was a truly collaborative effort. I am very happy to say that in my opinion we have a well automated and monitored Kubernetes cluster in Toolforge today. There are many more features that we will continue to work on as we try to make Kubernetes use in Toolforge easier for everyone, but we can only do that work because we now have this solid base to build on. I look forward to announcements of many more features in the coming months. Thank you to our alpha and beta testers who found more edge cases and made good suggestions for simplifying things. Thank you all for your patience and understanding when things did not go quite as planned during this process. And finally thank you in advance for the edits that will be made to help pages on Wikitech and elsewhere as we all work on bug #1 (improving documentation). [0]: https://phabricator.wikimedia.org/T246519 [1]: https://phabricator.wikimedia.org/T246689 [2]: https://phabricator.wikimedia.org/T214513 Bryan, on behalf of the Toolforge admin team and the Wikimedia Cloud Services team -- Bryan Davis Technical Engagement Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808

1 0

VM reboots on Thursday
by Andrew Bogott 04 Mar '20

04 Mar '20

Time for another hardware repair task! A lot of these fragile hosts are slated for replacement within the year but in the next time we need to nurse them along. On Friday I'm going to drain cloudvirt1006 for a battery replacement. Starting around 15:00 UTC the following VMs will be copied to another host, which means a short period of downtime and a reboot: phabricator-stage-1001 icinga-dev-01 tool-buster dwl-test deployment-echostore01 deployment-sessionstore03 dashiki-02 mwv-builder-03 images af-nb-fe traffic-ncredir deployment-logstash03 media-streaming bstorm-nfs-test maps-wma cvn-app9

1 1

Planned NFS maintenance 2020-02-26@1800 UTC
by Brooke Storm 26 Feb '20

26 Feb '20

On Wed 26-Feb, we are making a large change to how NFS is mounted in Cloud Services https://gerrit.wikimedia.org/r/c/operations/puppet/+/571821. This will impact any Cloud VPS projects that mount NFS for home directories, project directories and scratch, including Toolforge. During this change, NFS will become unresponsive for a short time. Some NFS clients will recover on their own with little impact. Where needed, WMCS will reboot or remount NFS clients. This change will improve future NFS management and hopefully reduce future disruptions from maintenance.

1 2

[Toolforge] 2020 Kubernetes cluster automatic migration phase beginning
by Bryan Davis 21 Feb '20

21 Feb '20

Following a beta testing period [0] and a general use self-migration period [1], the Toolforge administration team is ready to begin the final phase of automatic migration of tools currently running on the legacy Kubernetes cluster to the 2020 Kubernetes cluster. The migration process will involve Toolforge administrators running `webservice migrate` for each tool in the same way that self-migration happens [2]. A small number of tools are using the legacy Kubernetes cluster outside of the `webservice` system. These tools will be moved using a more manual process after move all webservices. We are currently planning on doing these migrations in several batches so that we can monitor the load and capacity of the 2020 Kubernetes cluster as we move ~640 more tools over from the legacy cluster. Once the tools have all been moved to the 2020 cluster, we will continue with additional clean up and default configuration changes which will allow us to fully decommission the legacy cluster. We will also be updating various documentation on Wikitech during this final phase. We hope to complete this entire process by 2020-03-06 at the latest. [0]: https://lists.wikimedia.org/pipermail/cloud-announce/2020-January/000247.ht… [1]: https://lists.wikimedia.org/pipermail/cloud-announce/2020-January/000252.ht… [2]: https://wikitech.wikimedia.org/wiki/News/2020_Kubernetes_cluster_migration#… Bryan (on behalf of the Toolforge admins and the Cloud Services team) -- Bryan Davis Technical Engagement Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808

1 0

VM reboots on Thursday
by Andrew Bogott 20 Feb '20

20 Feb '20

The barrage of hardware failures continues! Next week the eqiad staff will be repairing cloudvirt1014; to prepare I'll be draining it on this Thursday (2020-02-20), beginning around 15:00 UTC. Affected instances will be down for a few minutes and then rebooted. Toolforge users should be largely unaffected by this maintenance. Here is the list of affected VMs: traffic-cache-atsupload-buster canary1014-01 util-abogott-buster filippo-log-buster01 staging opusmt mw01 grantreview-04 cloud-puppetmaster-03 commtech-wikiwho toolsbeta-test-k8s-haproxy-2 debmonitor-pm toolsbeta-test-k8s-control-1 toolsbeta-test-k8s-etcd-2 toolsbeta-test-k8s-etcd-1 jmm-debm-01 xtools-dev05 ores-web-05 roebling ores-web-04 cloudinfra-db02 Krypton discovery-production-02 maps-tiles1 wikitextexp-base-1002 accounts-appserver4 tofawiki02 packagist-mirror1 deployment-elastic06 deployment-changeprop deployment-restbase02 deployment-imagescaler01 deployment-kafka-jumbo-1 deployment-memc07 deployment-eventlog05 deployment-cpjobqueue deployment-mediawiki-07 deployment-chromium01 deployment-cache-text05 whgi wikilabels gitservices wikilabels-02 af-puppetdb02 missing-sections ores-lb-03 matrix-synapse-01 captcha-tf-43 k4-2

1 1

Brief interruption in grid engine cronjobs tomorrow
by Andrew Bogott 20 Feb '20

20 Feb '20

Tomorrow I need to relocate 'tools-sgecron-01,' the VM that is in charge of starting cron jobs on the grid. The host will be down for 5-10 minutes, during which time no cronjobs will start. I'm going to make the move around 16:00 UTC, although I'll wait until a few minutes after so that on-the-hour jobs still happen. -Andrew

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-announce