Every year or so the Cloud Services team tries to identify and clean up
unused projects and VMs. We do this via an opt-in process: anyone can
mark a project as 'in use,' and that project will be preserved for
another year.
I've created a wiki page the lists all existing projects, here:
https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2019_Purge
If you are a VPS user, please visit that page and mark any projects that
you use as {{Used}}. Note that it's not necessary for you to be a
project admin to mark something -- if you know that you're currently
using a resource and want to keep using it, go ahead and mark it
accordingly. If you /are/ a project admin, please take a moment to mark
which VMs are or aren't used in your projects.
When December arrives, I will shut down and begin the process of
reclaiming resources from unused projects.
If you think you use a VPS project but aren't sure which, I encourage
you to poke around on https://tools.wmflabs.org/openstack-browser/ to
see what looks familiar. Worst case, just email
cloud(a)lists.wikimedia.org with a description of your use case and we'll
sort it out there.
Exclusive toolforge users are free to ignore this task.
Thank you!
-Andrew and WMCS team
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
Hi,
today 2019-09-30 we were doing an operation in all CloudVPS virtual machines to
update ferm to fix a bug [0]. Ferm is a firewalling utility.
The fleet-wide operation resulted in ferm being installed in every VM, even in
those VMs not requiring it. This resulted in a network outage for most of the
virtual machines and projects that were not previously configured to use ferm.
Many Toolforge tools (webservices, grid jobs, etc) stopped working, database
connection were lost, proxy reported bad gateway errors, etc.
To resolve the issue, we quickly removed ferm from every VM and run puppet agent
to install it just in the VMs that had ferm in their puppet manifests.
As soon as we did this, everything went back to normal.
This incident lasted 1h, give or take.
Please, get in contact in case you see any issue or have any doubts about this
incident.
regards.
[0] https://phabricator.wikimedia.org/T153468
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
The format of block expiry timestamps returned from list=users and
list=allusers has long been inconsistent. It was being returned in an
internal format (e.g. "20190918201053"), rather than the ISO 8601 format
used by the rest of the API (e.g. "2019-09-18T20:10:53Z").
The 'blockexpiry' property from these two modules will be changing to the
standard ISO 8601 format with 1.34.0-wmf.24. See
https://www.mediawiki.org/wiki/MediaWiki_1.34/Roadmap for a schedule of
deployment to Wikimedia wikis.
This change also brings the block information returned by list=users and
list=allusers fully in line with that already used for meta=userinfo and
'blocked' errors from various actions.
--
Brad Jorsch (Anomie)
Senior Software Engineer
Wikimedia Foundation
_______________________________________________
Mediawiki-api-announce mailing list
Mediawiki-api-announce(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce
Due to a mishap during routine data-center maintenance, one of our
hypervisors lost power just now. Everything is back up and running now,
but some of you may have experienced a few minutes of downtime and an
unexpected reboot of your instance.
Toolforge was largely unaffected by this incident, other than some jobs
getting rescheduled. The VMs that were restarted are:
accounts-dbslave.account-creation-assistance.eqiad.wmflabs
af-netbox01.automation-framework.eqiad.wmflabs
arturo-k8s-test-2.openstack.eqiad.wmflabs
arturo-k8s-test-3.openstack.eqiad.wmflabs
arturo-k8s-test-4-2.openstack.eqiad.wmflabs
beryllium.rcm.eqiad.wmflabs
canary1027-01.testlabs.eqiad.wmflabs
captcha-imageprocessing-11.privpol-captcha.eqiad.wmflabs
clouddb-services-puppetmaster-01.clouddb-services.eqiad.wmflabs
deployment-acme-chief04.deployment-prep.eqiad.wmflabs
deployment-aqs01.deployment-prep.eqiad.wmflabs
deployment-aqs02.deployment-prep.eqiad.wmflabs
deployment-db06.deployment-prep.eqiad.wmflabs
deployment-prometheus02.deployment-prep.eqiad.wmflabs
gnd-02.orig.eqiad.wmflabs
jbond-buster.puppet.eqiad.wmflabs
krenair-t219424-b.testlabs.eqiad.wmflabs
lizenzhinweisgenerator-api-test.lizenzhinweisgenerator.eqiad.wmflabs
logstack03.security-tools.eqiad.wmflabs
mcr-sdc.mcr-dev.eqiad.wmflabs
ntp-02.cloudinfra.eqiad.wmflabs
paws-int-lb-02.paws.eqiad.wmflabs
paws-master-02.paws.eqiad.wmflabs
paws-packages-01.paws.eqiad.wmflabs
paws-proxy-02.paws.eqiad.wmflabs
paws-puppetmaster-01.paws.eqiad.wmflabs
paws-worker-01.paws.eqiad.wmflabs
proxy-01.project-proxy.eqiad.wmflabs
redirects-nginx01.redirects.eqiad.wmflabs
sentry-builder.sentry.eqiad.wmflabs
toolsbeta-docker-registry-01.toolsbeta.eqiad.wmflabs
wikibase-stretch.wikidata-dev.eqiad.wmflabs
wpx-mediawiki-02.wpx.eqiad.wmflabs
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
On June 30, 2020 the Debian project will stop providing security patch
support for the Debian 8 "Jessie" release. The Cloud Services and SRE
teams at the Wikimedia Foundation would like to have all usage of
Debian Jessie in our managed networks replaced with newer versions of
Debian's operating system on or ideally well before that date.
A page has been created on Wikitech [0] with an initial timeline for
the removal of all Debian Jessie instances from Cloud VPS projects.
This timeline follows roughly the same schedule as we used in 2018
when deprecating Ubuntu Trusty in Cloud VPS projects:
* September 2019: Announce the initiative via this email and the Wikitech page
* October 2019: Start actively contacting instance maintainers who
need to migrate to a new OS
* November & December 2019: Continue to work with instance maintainers
to migrate to a new OS
* January 2020: Shutdown remaining Debian Jessie instances
If you know that your Cloud VPS project is using Debian Jessie, you
can get a head start on migrating your instances to Debian Buster
(preferred) or Stretch by visiting the Wikitech page and reading the
instructions there.
If you are a concerned Toolforge user, stay tuned for future
announcements about changes that will be made as the Toolforge admin
team works to remove Debian Jessie from that environment. For now
there is nothing an individual Tool maintainer needs to do.
[0]: https://wikitech.wikimedia.org/wiki/News/Jessie_deprecation
Bryan - on behalf of the Cloud VPS admin team
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
Cross-posting Johan's message from wikitech-l to signal boost. This is
a long survey if you fill in all the questions, but it really does
help the Wikimedia Foundation and others select and prioritize work.
Bryan
---------- Forwarded message ---------
From: Johan Jönsson <jjonsson(a)wikimedia.org>
Date: Wed, Sep 11, 2019 at 11:15 AM
Subject: [Wikitech-l] The Community Insights survey (for volunteer devs)
To: Wikimedia developers <wikitech-l(a)lists.wikimedia.org>
Hey folks, I'm helping Rebecca Maung (rmaung(a)wikimedia.org) distribute
this request. Her words below:
The Wikimedia Foundation is asking for your feedback in the annual
Community Insights survey. We want to know how well we are supporting
your work on- and off-wiki, and how we can change or improve things in
the future. The opinions you share will directly affect the current
and future work of the Wikimedia Foundation.
If you are a volunteer developer, and have contributed code to any
pieces of MediaWiki, gadgets, or tools, please complete the survey. It
is available in various languages and will take between 15 and 25
minutes to complete.
Follow this link to the survey:
https://wikimedia.qualtrics.com/jfe/form/SV_0pSrrkJAKVRXPpj?Target=dev
If you have seen a similar message elsewhere and have already taken
the Community Insights survey, please do not take it twice.
You can find more information about this survey on the project page
and see how your feedback helps the Wikimedia Foundation support
contributors like you. This survey is hosted by a third-party service
and governed by this privacy statement. Please visit our frequently
asked questions page to find more information about this survey.
If you need additional help, send an email to surveys(a)wikimedia.org.
Thank you!
//Johan Jönsson
--
_______________________________________________
Wikitech-l mailing list
Wikitech-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
(Note: This is only an early heads-up, to be prepared. Google Code-in
has NOT been announced yet, but last year, GCI mentors asked for more
time in advance to identify tasks to mentor. Here you are. :)
* You have small, self-contained bugs you'd like to see fixed?
* Your documentation needs specific improvements?
* Your user interface has some smaller design issues?
* Your Outreachy/Summer of Code project welcomes small tweaks?
* You'd enjoy helping someone port your template to Lua?
* Your gadget code uses some deprecated API calls?
* You have tasks in mind that welcome some research?
Google Code-in (GCI) is an annual contest for 13-17 year old students.
GCI 2019 has not yet been announced but usually takes place from late
October to December. It is not only about coding: We also need tasks
about design, docs, outreach/research, QA.
Read https://www.mediawiki.org/wiki/Google_Code-in/Mentors , add
your name to the mentors table, and start tagging tasks in Wikimedia
Phabricator by adding the #gci-2019 project tag.
We will need MANY mentors and MANY tasks, otherwise we cannot make it.
Last year, 199 students successfully worked on 765 tasks supported by
39 mentors. For some achievements from the last round, see
https://wikimediafoundation.org/news/2019/02/20/partnerships-make-it-possib…
Note that "beginner tasks" (e.g. "Set up Vagrant") and generic
tasks are very welcome (like "Choose and replace 2 uses of
Linker::link() from the list in T223010" style).
We also have more than 400 unassigned open #good-first-bug tasks:
https://phabricator.wikimedia.org/maniphest/query/3YnDUWYJfXSo/#R
Can and would you mentor some of these tasks in your area?
Please take a moment to find / update [Phabricator etc.] tasks in your
project(s) which would take an experienced contributor 2-3 hours. Read
https://www.mediawiki.org/wiki/Google_Code-in/Mentors
, ask if you have any questions, and add your name to
https://www.mediawiki.org/wiki/Google_Code-in/2019#List_of_Wikimedia_mentors
Thanks (as we will not be able to run this without your help),
andre
--
Andre Klapper (he/him) | Bugwrangler / Developer Advocate
https://blogs.gnome.org/aklapper/
We need to do a proper incident report, but I wanted to send out a
(late) notice that the Toolforge Kubernetes cluster was at best
degraded and at worst completely broken from 2019-09-10T18:54 to
2019-09-11T01:30.
The TL;DR is that some change, likely part of T171188: Move the main
WMCS puppetmaster into the Labs realm, tricked Puppet into installing
an old version of the x509 signing cert used to secure communication
between the etcd cluster and kube-apiserver. This manifested in an
alert from our monitoring system of the Kubernetes api being broken.
When investigating that alert we found that the kube-apiserver was
unable to connect to its paired etcd cluster. The etcd cluster seemed
to be flapping internally (status showing good, then failed, then good
again). Diagnosing the cause of this flapping resulted in a complete
failure of the etcd cluster. Restoring the etcd cluster was a long and
difficult task. Once etcd was recovered, it took about 1.5 more hours
to find the cause and fix for the initial communication errors (the
wrong x509 signing certificate). It is currently unclear if the x509
misconfiguration also caused the etcd cluster failure, or if that was
an unrelated and unfortunate coincidence.
See https://phabricator.wikimedia.org/T232536 for follow up
documentation (when we write it during the coming US business day).
Bryan - on behalf of the Toolforge admin team
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
When saving an edit is prevented by the AbuseFilter or SpamBlacklist
extensions, the error is currently reported as a successful API response
with a 'failure' code in the body.[1][2]
In the future, these will be reported as standard API errors.[3][4]
This change should be deployed to Wikimedia wikis with 1.34.0-wmf.23. See
https://www.mediawiki.org/wiki/MediaWiki_1.34/Roadmap for a schedule.
Clients that do not need to specially handle failures due to AbuseFilter or
SpamBlacklist will likely need no changes, as they probably already include
code to generically handle API error responses.
Clients that do handle AbuseFilter or SpamBlacklist failures specially will
need to be updated to check for error codes 'abusefilter-warning',
'abusefilter-disallowed', and/or 'spamblacklist' and handle them as they do
the current AbuseFilter and SpamBlacklist failures, if they want to
preserve their current special handling.
Note that edit failures due to CAPTCHAs from ConfirmEdit are not being
changed at this time. They will continue to be reported as before.[5]
[1]: AbuseFilter: https://phabricator.wikimedia.org/P8988
[2]: SpamBlacklist: https://phabricator.wikimedia.org/P8990
[3]: AbuseFilter: https://phabricator.wikimedia.org/P8989
[4]: SpamBlacklist: https://phabricator.wikimedia.org/P8991
[5]: ConfirmEdit: https://phabricator.wikimedia.org/P9076
--
Brad Jorsch (Anomie)
Senior Software Engineer
Wikimedia Foundation
_______________________________________________
Mediawiki-api-announce mailing list
Mediawiki-api-announce(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce
Later today (starting in a few hours around 18:00 UTC) we'll be
rearranging the puppetmaster setup for most cloud VMs[0]. No tools or
services (other than puppet) should be affected, but some of you might
get grumpy emails about broken puppet runs during the transition, which
I encourage you to ignore. If you're planning to update the puppet
configuration of your VMs, I encourage you to postpone that work until
after our migration.
[0] full context at https://phabricator.wikimedia.org/T171188
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce