We'll be upgrading the cloud services OpenStack install tomorrow,
beginning at 15:00 UTC.
There should be little to no interruption to VMs or Toolforge, but
Horizon logins will be disabled for part of the window.
Sorry for the short notice!
- Andrew + the WMCS team
Hi there!
If you use a CloudVPS web proxy, this email is for you. Toolforge
developers/users can ignore this email.
We are introducing a change to eliminate the 'X-Forwarded-For' HTTP header that
the CloudVPS web proxy adds when forwarding the HTTP request to your instance.
This header contains the original IP address of the internet client that sent
the request. This is private information that we would like to reduce in our
environment [0].
You use the web proxy if you have a public web endpoint hosted in CloudVPS under
the wmflabs.org domain. These are generally configured using Horizon in the DNS
> Web Proxies section.
Examples of web proxy names:
* accounts.wmflabs.org
* glampipe.wmflabs.org
* incubator.wmflabs.org
Full list can be seen in the Openstack Browser tool [1].
We are ready to introduce this change [2], but wanted to give some heads up for
projects that do require this information for whatever reason. We would like to
hear from you in the next couple of weeks. Please contact us in the phabricator
task [0] and include some rationale why you need the XFF header.
This is the timeline this change will follow:
* 2020-04-01: this email, start collecting list of things that require XFF
* 2020-04-07: start evaluating list of things that require XFF
* 2020-04-15: introduce the change, with proper case whitelisting
When the change is introduced, in two weeks from now, proxy backends that were
not whitelisted will stop receiving the XFF header.
Please reach out for any questions or comments.
regards.
[0] https://phabricator.wikimedia.org/T135046
[1] https://openstack-browser.toolforge.org/project/project-proxy
[2] https://gerrit.wikimedia.org/r/c/operations/puppet/+/583098
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
Hi there!
In a few days from now (2020-04-13), the CloudVPS network will see a change
happening that will likely go unnoticed, but it is important enough to share it
with you beforehand.
We will be changing the IPv4 address that we use as the main source NAT for
egress connections (initiated in the VM instances). This change won't affect VM
instances using floating IPs.
Old IP address: 185.15.56.1
New IP address: 208.80.155.92
If you know of anywhere (a firewall, ACL or any other mechanism) that had this
address hardcoded, you will need to update it.
See this wikitech page for more details:
https://wikitech.wikimedia.org/wiki/News/CloudVPS_NAT_change
Please reach out if you have any doubts, questions, or any other issue.
regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
Next Thursday we'll be upgrading to OpenStack version 'Queens' starting
at 15:00 UTC.
The upgrade will take a couple of hours. During the upgrade process,
Horizon (and associated OpenStack APIs) will be disabled. There may
also be brief network interruptions during the upgrade.
Toolforge and existing VMs should be largely unaffected if all goes well
-- there may be some service interruption if we encounter networking issues.
- Andrew + the WMCS team
Hello all,
For almost a year, the Wikidata development team has been working on the
task of redesigning and migrating the wb_terms table, which had become too
big and unsustainable over the years.
You can read the tale of our journey on this blog post: Come to Terms with
Changes
<https://phabricator.wikimedia.org/phame/post/view/195/coming_to_terms_with_…>
If you’re a tool maintainer and your tool queries directly the Labs
database replicas, you can read more details
<https://lists.wikimedia.org/pipermail/wikidata/2020-March/013901.html>
about the next steps and how to update your code.
Congratulations to all the developers involved in this big project!
Cheers,
--
Léa Lacroix
Project Manager Community Communication for Wikidata
Wikimedia Deutschland e.V.
Tempelhofer Ufer 23-24
10963 Berlin
www.wikimedia.de
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/029/42207.
On 2020-03-03, Brooke completed the automatic migration phase of the
2020 Kubernetes migration by moving the last workloads from the legacy
Kubernetes cluster to the 2020 Kubernetes cluster [0].
All Toolforge tools using `webservice --backend=kubernetes ...` and/or
manually maintained Kubernetes objects are now running on the 2020
Kubernetes cluster. The Toolforge admin team is in the process of
tearing down the legacy cluster and cleaning up various documentation
and tooling related to it [1].
This project involved a lot of hard work that most of the Toolforge
community did not see. Brooke and Arturo started planning things over
a year ago [2] to ensure that the Toolforge admin team would be able
to complete this migration with a minimum amount of disruption to
tools and their maintainers. Along the journey they researched
Kubernetes best practices and recommendations, read and re-read
numerous tutorial and how-to docs, and designed a completely new
process to automate the deployment of Kubernetes in Toolforge. They
also sought and received help from other Toolforge admins, Wikimedia
Foundation staff, and technical volunteers. This was a truly
collaborative effort.
I am very happy to say that in my opinion we have a well automated and
monitored Kubernetes cluster in Toolforge today. There are many more
features that we will continue to work on as we try to make Kubernetes
use in Toolforge easier for everyone, but we can only do that work
because we now have this solid base to build on. I look forward to
announcements of many more features in the coming months.
Thank you to our alpha and beta testers who found more edge cases and
made good suggestions for simplifying things. Thank you all for your
patience and understanding when things did not go quite as planned
during this process. And finally thank you in advance for the edits
that will be made to help pages on Wikitech and elsewhere as we all
work on bug #1 (improving documentation).
[0]: https://phabricator.wikimedia.org/T246519
[1]: https://phabricator.wikimedia.org/T246689
[2]: https://phabricator.wikimedia.org/T214513
Bryan, on behalf of the Toolforge admin team and the Wikimedia Cloud
Services team
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
Time for another hardware repair task! A lot of these fragile hosts are
slated for replacement within the year but in the next time we need to
nurse them along.
On Friday I'm going to drain cloudvirt1006 for a battery replacement.
Starting around 15:00 UTC the following VMs will be copied to another
host, which means a short period of downtime and a reboot:
phabricator-stage-1001
icinga-dev-01
tool-buster
dwl-test
deployment-echostore01
deployment-sessionstore03
dashiki-02
mwv-builder-03
images
af-nb-fe
traffic-ncredir
deployment-logstash03
media-streaming
bstorm-nfs-test
maps-wma
cvn-app9
On Wed 26-Feb, we are making a large change to how NFS is mounted in
Cloud Services
https://gerrit.wikimedia.org/r/c/operations/puppet/+/571821. This will
impact any Cloud VPS projects that mount NFS for home directories,
project directories and scratch, including Toolforge. During this
change, NFS will become unresponsive for a short time. Some NFS clients
will recover on their own with little impact. Where needed, WMCS will
reboot or remount NFS clients.
This change will improve future NFS management and hopefully reduce
future disruptions from maintenance.
Following a beta testing period [0] and a general use self-migration
period [1], the Toolforge administration team is ready to begin the
final phase of automatic migration of tools currently running on the
legacy Kubernetes cluster to the 2020 Kubernetes cluster.
The migration process will involve Toolforge administrators running
`webservice migrate` for each tool in the same way that self-migration
happens [2]. A small number of tools are using the legacy Kubernetes
cluster outside of the `webservice` system. These tools will be moved
using a more manual process after move all webservices. We are
currently planning on doing these migrations in several batches so
that we can monitor the load and capacity of the 2020 Kubernetes
cluster as we move ~640 more tools over from the legacy cluster.
Once the tools have all been moved to the 2020 cluster, we will
continue with additional clean up and default configuration changes
which will allow us to fully decommission the legacy cluster. We will
also be updating various documentation on Wikitech during this final
phase. We hope to complete this entire process by 2020-03-06 at the
latest.
[0]: https://lists.wikimedia.org/pipermail/cloud-announce/2020-January/000247.ht…
[1]: https://lists.wikimedia.org/pipermail/cloud-announce/2020-January/000252.ht…
[2]: https://wikitech.wikimedia.org/wiki/News/2020_Kubernetes_cluster_migration#…
Bryan (on behalf of the Toolforge admins and the Cloud Services team)
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
The barrage of hardware failures continues! Next week the eqiad staff
will be repairing cloudvirt1014; to prepare I'll be draining it on this
Thursday (2020-02-20), beginning around 15:00 UTC. Affected instances
will be down for a few minutes and then rebooted. Toolforge users should
be largely unaffected by this maintenance.
Here is the list of affected VMs:
traffic-cache-atsupload-buster
canary1014-01
util-abogott-buster
filippo-log-buster01
staging
opusmt
mw01
grantreview-04
cloud-puppetmaster-03
commtech-wikiwho
toolsbeta-test-k8s-haproxy-2
debmonitor-pm
toolsbeta-test-k8s-control-1
toolsbeta-test-k8s-etcd-2
toolsbeta-test-k8s-etcd-1
jmm-debm-01
xtools-dev05
ores-web-05
roebling
ores-web-04
cloudinfra-db02
Krypton
discovery-production-02
maps-tiles1
wikitextexp-base-1002
accounts-appserver4
tofawiki02
packagist-mirror1
deployment-elastic06
deployment-changeprop
deployment-restbase02
deployment-imagescaler01
deployment-kafka-jumbo-1
deployment-memc07
deployment-eventlog05
deployment-cpjobqueue
deployment-mediawiki-07
deployment-chromium01
deployment-cache-text05
whgi
wikilabels
gitservices
wikilabels-02
af-puppetdb02
missing-sections
ores-lb-03
matrix-synapse-01
captcha-tf-43
k4-2