On Wed 26-Feb, we are making a large change to how NFS is mounted in
Cloud Services
https://gerrit.wikimedia.org/r/c/operations/puppet/+/571821. This will
impact any Cloud VPS projects that mount NFS for home directories,
project directories and scratch, including Toolforge. During this
change, NFS will become unresponsive for a short time. Some NFS clients
will recover on their own with little impact. Where needed, WMCS will
reboot or remount NFS clients.
This change will improve future NFS management and hopefully reduce
future disruptions from maintenance.
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
Following a beta testing period [0] and a general use self-migration
period [1], the Toolforge administration team is ready to begin the
final phase of automatic migration of tools currently running on the
legacy Kubernetes cluster to the 2020 Kubernetes cluster.
The migration process will involve Toolforge administrators running
`webservice migrate` for each tool in the same way that self-migration
happens [2]. A small number of tools are using the legacy Kubernetes
cluster outside of the `webservice` system. These tools will be moved
using a more manual process after move all webservices. We are
currently planning on doing these migrations in several batches so
that we can monitor the load and capacity of the 2020 Kubernetes
cluster as we move ~640 more tools over from the legacy cluster.
Once the tools have all been moved to the 2020 cluster, we will
continue with additional clean up and default configuration changes
which will allow us to fully decommission the legacy cluster. We will
also be updating various documentation on Wikitech during this final
phase. We hope to complete this entire process by 2020-03-06 at the
latest.
[0]: https://lists.wikimedia.org/pipermail/cloud-announce/2020-January/000247.ht…
[1]: https://lists.wikimedia.org/pipermail/cloud-announce/2020-January/000252.ht…
[2]: https://wikitech.wikimedia.org/wiki/News/2020_Kubernetes_cluster_migration#…
Bryan (on behalf of the Toolforge admins and the Cloud Services team)
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
The bastion hosts don't have python3.7, and the k8s hosts don't have ssh, so "git pull" and "git push" don't work. Which means I need to keep flipping back and forth between the two during development. Is there some way to get ssh working on the k8s hosts, so I can do everything in one place?
The other annoyance is that when you connect with "webservice --backend=kubernetes python3.7 shell", the connection eats control-p's Is there some way to turn that off, or at least reconfigure it to use a different command character?
The barrage of hardware failures continues! Next week the eqiad staff
will be repairing cloudvirt1014; to prepare I'll be draining it on this
Thursday (2020-02-20), beginning around 15:00 UTC. Affected instances
will be down for a few minutes and then rebooted. Toolforge users should
be largely unaffected by this maintenance.
Here is the list of affected VMs:
traffic-cache-atsupload-buster
canary1014-01
util-abogott-buster
filippo-log-buster01
staging
opusmt
mw01
grantreview-04
cloud-puppetmaster-03
commtech-wikiwho
toolsbeta-test-k8s-haproxy-2
debmonitor-pm
toolsbeta-test-k8s-control-1
toolsbeta-test-k8s-etcd-2
toolsbeta-test-k8s-etcd-1
jmm-debm-01
xtools-dev05
ores-web-05
roebling
ores-web-04
cloudinfra-db02
Krypton
discovery-production-02
maps-tiles1
wikitextexp-base-1002
accounts-appserver4
tofawiki02
packagist-mirror1
deployment-elastic06
deployment-changeprop
deployment-restbase02
deployment-imagescaler01
deployment-kafka-jumbo-1
deployment-memc07
deployment-eventlog05
deployment-cpjobqueue
deployment-mediawiki-07
deployment-chromium01
deployment-cache-text05
whgi
wikilabels
gitservices
wikilabels-02
af-puppetdb02
missing-sections
ores-lb-03
matrix-synapse-01
captcha-tf-43
k4-2
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
Tomorrow I need to relocate 'tools-sgecron-01,' the VM that is in charge
of starting cron jobs on the grid. The host will be down for 5-10
minutes, during which time no cronjobs will start.
I'm going to make the move around 16:00 UTC, although I'll wait until a
few minutes after so that on-the-hour jobs still happen.
-Andrew
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
Hello,
FYI, I know most of you may have no idea what wm-bot is, but I don't
know any better channel to use for this kind of announcement so, I had
to move wm-bot to a new instance as old one was using deprecated OS,
which means, that anyone who is using its "netcat" feature - relaying
TCP messages directly to IRC channels
https://meta.wikimedia.org/wiki/Wm-bot#Relay_of_messages_from_scripts_and_o…
will have to change the IP / hostname.
To avoid this kind of unnecessary work in the future: is it possible
to create some VIP in cloud environment so that I can simply move this
address to a new instance when I perform this kind of migration? Also,
as many users are using the hostname instead of IP, what's the best
way to move the hostname? Would it be possible to implement some
instance renaming feature?
Thanks
On Tuesday morning I'm going to switch the OpenStack Keystone token
engine from UUID tokens to fernet tokens[0]. The changeover[1] will be
abrupt and cause all existing sessions to reset (e.g. if you're using
Horizon you'll have to log back in, and if you're in the middle of
creating a VM that creation will probably fail.) Should you encounter
this interruption, just try again in a few minutes and everything should
be fine.
The new tokens will generally behave the same from a user standpoint,
but will allow us to simplify and modernize things on our end a bit.
The switch will happen at 15:00 UTC on this Tuesday, 2020-02-18. That's
7AM, Pacific Coast time.
[0]
https://docs.openstack.org/keystone/pike/admin/identity-fernet-token-faq.ht…
[1] https://phabricator.wikimedia.org/T243418
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
In order to repair some ailing hardware, I'm going to migrate several
cloud-vps instances later today. Each will be down for a few minutes
(or longer, depending on disk size) and then rebooted. Toolforge users
will be unaffected by this change.
Affected VMs will be:
vanadium
cloudstore-client-2
cloudstore-client-1
cn-staging-2
wikispore-test
wm-bot
traffic-cache-atstext-buster
etherpad
traffic-cache-atsupload
jbond-buster
commons-corruption-checker-main
codesearch6
canary1022-01
ores-web-06
cloud-cumin-01
petscan4
thanos-prom01
thanos-be01
extdist-04
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
*(feel free to forward the message as is to your friends, family members &
colleagues)*
Hello folks,
We would like to invite you to apply to the Outreachy and Google Summer of
Code program with the Wikimedia Foundation (a non-profit organization
behind Wikipedia)!
*About the Outreachy program*
Wikimedia will be mentoring ~7 projects in the Outreachy program in the May
to August 2020 round. The initial applications are due February 25th at 4
pm UTC.
Apply today: https://www.outreachy.org/apply/
Outreachy offers three-month internships to work remotely in Free and Open
Source Software (FOSS) projects with experienced mentors. The internships
may include programming, user experience, documentation, illustration and
graphic design, or data science.
Outreachy internships run twice a year – from May to August and December to
March. Interns are paid a stipend of $5,500 USD for the three months of
work. They also have a $500 USD stipend to travel to conferences and
events. Interns often find employment after their internship with Outreachy
sponsors or in jobs that use the skills they learned during their
internship.
Outreachy is open to both students and non-students. Outreachy expressly
invites the following people to apply:
* Women (both cis and trans), trans men, and genderqueer people.
* Anyone who faces under-representation, systematic bias, or discrimination
in the technology industry in their country or residence is invited to
apply.
* Residents and nationals of the United States of any gender who are
Black/African American, Hispanic/Latin@, Native American/American Indian,
Alaska Native, Native Hawaiian, or Pacific Islander.
*About the Google Summer of Code program*
Wikimedia is planning to mentor 8-10 projects in 2020’s Google Summer of
Code (GSoC) program. Beginning February 20th, pending Wikimedia’s
acceptance as a mentoring organization, applicants can begin discussing
ideas with the mentors!
The student application will be due on March 31st 18:00 UTC:
https://summerofcode.withgoogle.com/
Google Summer of Code now in its 16th year, is Google's summer program for
university students who want to get involved in open source software. Over
15,960 students from 109 countries have already participated. Google Summer
of Code is a unique program that pairs students with mentors who introduce
them to the open-source community and provide guidance while they work on
real-world open-source projects during their summer break from university.
Projects cover a wide range of fields including: Cloud, Operating Systems,
Graphics, Medicine, Programming Languages, Robotics, Science, Security and
many more. It's a highly competitive program (and this year is expected to
be even bigger than last year), so don't wait until the last minute to
prepare!
*About the Wikimedia Foundation*
The Wikimedia Foundation <https://wikimediafoundation.org/> is the
nonprofit organization that hosts and operates Wikipedia and the other
Wikimedia
free knowledge projects
<https://wikimediafoundation.org/our-work/wikimedia-projects/>. Our vision
is a world in which every single human can freely share in the sum of all
knowledge. We believe that everyone has the potential to contribute
something to our shared knowledge, and that everyone should be able to
access that knowledge, free of interference. We host the Wikimedia
projects, build software experiences for reading, contributing, and sharing
Wikimedia content, support the volunteer communities and partners who make
Wikimedia possible, and advocate for policies that enable Wikimedia and
free knowledge to thrive.
*Resources*
* Browse through the participants’ guides, to learn more about the
application process steps:
https://www.mediawiki.org/wiki/Outreachy/Participantshttps://www.mediawiki.org/wiki/Google_Summer_of_Code/Participants
* All the projects will be showcased here:
https://www.outreachy.org/communities/cfp/wikimedia/https://www.mediawiki.org/wiki/Google_Summer_of_Code/2020
We hope you will help us spread the word about Wikimedia’s participation in
these programs: https://twitter.com/mediawiki/status/1224790357361098752
(by retweeting the post in the link or by sharing this email)
Looking forward to your participation!
Cheers,
Srishti & Pavithra (Wikimedia organization administrators for GSoC &
Outreachy)
*Srishti Sethi*
Developer Advocate
Wikimedia Foundation <https://wikimediafoundation.org/>