The <wikidb>.analytics.db.svc.eqiad.wmflabs database servers have been
experiencing some stability issues in the last two to three weeks that
we have reason to believe are related to query volume. The DBA team at
the Wikimedia Foundation is looking into various changes that may help
with these problems including software upgrades for our MariaDB
deployments.
Today we took an initial step of reducing the maximum time allowed for
a query to complete on the <wikidb>.analytics.db.svc.eqiad.wmflabs
hosts to 1 hour. We were using an upper limit of 4 hours previously.
Our hope is that this change will relieve some stress on the shared
servers and allow us more time to look into other changes to restore
stability. Ideally we will be able to increase the limit again after
making other changes to these systems.
Bryan, on behalf of the Cloud Services team
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
We'll be upgrading the cloud services OpenStack install on Monday,
beginning at 14:00 UTC.
The entire upgrade process may take a couple of hours. Early on in the
process, Horizon (and associated OpenStack APIs) will be disabled
(probably for 20 to 30 minutes.) There may also be brief network
interruptions during the upgrade, although if all goes well these will
not be noticeable by users.
Toolforge and existing VMs should be largely unaffected apart from
possible network hiccups.
- Andrew + the WMCS team
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
We had issues with NFS mounts in Toolforge as an unintended side
effect of OpenStack upgrades earlier today. While NFS mounts were
broken, many (all?) cronjobs failed. The emails telling maintainers
about these failures did not go out in real time because the mail
system also relies on NFS. When we fixed the mail server thousands of
queued messages started going out. There are still a few emails
sending as I write this, but the queue is almost empty.
As far as we can tell, cronjobs are starting as expected now and the
emails are just left over noise.
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
Hi Everyone,
We’re happy to announce the launch of the brand new quarterly Technical
Community Newsletter
<https://www.mediawiki.org/wiki/Technical_Community_Newsletter>. The
newsletter is compiled by the Wikimedia Developer Advocacy Team. It aims to
share highlights, news, and information of interest from and about the
Wikimedia technical community.
The First Edition of the Technical Community Newsletter
<https://www.mediawiki.org/wiki/Technical_Community_Newsletter/2019/October>
launches today! Find out more about what technical contributors have been
up to this past quarter, upcoming conferences & calls for papers, and how
to get involved.
Like all good things, the first edition starts as a prototype. Some ideas
we had in mind for additional sections/categories are “Tool of the Season”,
“Wikimedia in numbers” (i.e. around code review, new developer accounts,
etc). Ideas about what you would like to see and highlights you would like
to share are welcome
<https://www.mediawiki.org/wiki/Talk:Technical_Community_Newsletter>!
Subscribe to the Technical Community Newsletter
<https://www.mediawiki.org/wiki/Technical_Community_Newsletter/Subscribe>,
if you'd like to keep up with essential updates and information.
Kindly,
Sarah R. Rodlund
Technical Writer, Developer Advocacy
<https://meta.wikimedia.org/wiki/Developer_Advocacy>
srodlund(a)wikimedia.org
Hi all,
action=helppanelquestionposter is an API meant for internal use by the
GrowthExperiments extension's web UI, but it has not been formally marked
as internal, so I'm announcing the recent changes [1] to it just in case:
- the 'email' and 'resendconfirmation' parameters have been removed;
- the 'email' response field has been removed;
- the API is now marked as internal.
The functionality provided by the API is not really useful outside of the
web-based workflow it supports, and can be approximated with other, public
APIs, so this is not expected to cause problems to any clients.
[1]
https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/5…
_______________________________________________
Mediawiki-api-announce mailing list
Mediawiki-api-announce(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce
Hi,
today 2019-09-30 we were doing an operation in all CloudVPS virtual machines to
update ferm to fix a bug [0]. Ferm is a firewalling utility.
The fleet-wide operation resulted in ferm being installed in every VM, even in
those VMs not requiring it. This resulted in a network outage for most of the
virtual machines and projects that were not previously configured to use ferm.
Many Toolforge tools (webservices, grid jobs, etc) stopped working, database
connection were lost, proxy reported bad gateway errors, etc.
To resolve the issue, we quickly removed ferm from every VM and run puppet agent
to install it just in the VMs that had ferm in their puppet manifests.
As soon as we did this, everything went back to normal.
This incident lasted 1h, give or take.
Please, get in contact in case you see any issue or have any doubts about this
incident.
regards.
[0] https://phabricator.wikimedia.org/T153468
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
The format of block expiry timestamps returned from list=users and
list=allusers has long been inconsistent. It was being returned in an
internal format (e.g. "20190918201053"), rather than the ISO 8601 format
used by the rest of the API (e.g. "2019-09-18T20:10:53Z").
The 'blockexpiry' property from these two modules will be changing to the
standard ISO 8601 format with 1.34.0-wmf.24. See
https://www.mediawiki.org/wiki/MediaWiki_1.34/Roadmap for a schedule of
deployment to Wikimedia wikis.
This change also brings the block information returned by list=users and
list=allusers fully in line with that already used for meta=userinfo and
'blocked' errors from various actions.
--
Brad Jorsch (Anomie)
Senior Software Engineer
Wikimedia Foundation
_______________________________________________
Mediawiki-api-announce mailing list
Mediawiki-api-announce(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce
Due to a mishap during routine data-center maintenance, one of our
hypervisors lost power just now. Everything is back up and running now,
but some of you may have experienced a few minutes of downtime and an
unexpected reboot of your instance.
Toolforge was largely unaffected by this incident, other than some jobs
getting rescheduled. The VMs that were restarted are:
accounts-dbslave.account-creation-assistance.eqiad.wmflabs
af-netbox01.automation-framework.eqiad.wmflabs
arturo-k8s-test-2.openstack.eqiad.wmflabs
arturo-k8s-test-3.openstack.eqiad.wmflabs
arturo-k8s-test-4-2.openstack.eqiad.wmflabs
beryllium.rcm.eqiad.wmflabs
canary1027-01.testlabs.eqiad.wmflabs
captcha-imageprocessing-11.privpol-captcha.eqiad.wmflabs
clouddb-services-puppetmaster-01.clouddb-services.eqiad.wmflabs
deployment-acme-chief04.deployment-prep.eqiad.wmflabs
deployment-aqs01.deployment-prep.eqiad.wmflabs
deployment-aqs02.deployment-prep.eqiad.wmflabs
deployment-db06.deployment-prep.eqiad.wmflabs
deployment-prometheus02.deployment-prep.eqiad.wmflabs
gnd-02.orig.eqiad.wmflabs
jbond-buster.puppet.eqiad.wmflabs
krenair-t219424-b.testlabs.eqiad.wmflabs
lizenzhinweisgenerator-api-test.lizenzhinweisgenerator.eqiad.wmflabs
logstack03.security-tools.eqiad.wmflabs
mcr-sdc.mcr-dev.eqiad.wmflabs
ntp-02.cloudinfra.eqiad.wmflabs
paws-int-lb-02.paws.eqiad.wmflabs
paws-master-02.paws.eqiad.wmflabs
paws-packages-01.paws.eqiad.wmflabs
paws-proxy-02.paws.eqiad.wmflabs
paws-puppetmaster-01.paws.eqiad.wmflabs
paws-worker-01.paws.eqiad.wmflabs
proxy-01.project-proxy.eqiad.wmflabs
redirects-nginx01.redirects.eqiad.wmflabs
sentry-builder.sentry.eqiad.wmflabs
toolsbeta-docker-registry-01.toolsbeta.eqiad.wmflabs
wikibase-stretch.wikidata-dev.eqiad.wmflabs
wpx-mediawiki-02.wpx.eqiad.wmflabs
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
On June 30, 2020 the Debian project will stop providing security patch
support for the Debian 8 "Jessie" release. The Cloud Services and SRE
teams at the Wikimedia Foundation would like to have all usage of
Debian Jessie in our managed networks replaced with newer versions of
Debian's operating system on or ideally well before that date.
A page has been created on Wikitech [0] with an initial timeline for
the removal of all Debian Jessie instances from Cloud VPS projects.
This timeline follows roughly the same schedule as we used in 2018
when deprecating Ubuntu Trusty in Cloud VPS projects:
* September 2019: Announce the initiative via this email and the Wikitech page
* October 2019: Start actively contacting instance maintainers who
need to migrate to a new OS
* November & December 2019: Continue to work with instance maintainers
to migrate to a new OS
* January 2020: Shutdown remaining Debian Jessie instances
If you know that your Cloud VPS project is using Debian Jessie, you
can get a head start on migrating your instances to Debian Buster
(preferred) or Stretch by visiting the Wikitech page and reading the
instructions there.
If you are a concerned Toolforge user, stay tuned for future
announcements about changes that will be made as the Toolforge admin
team works to remove Debian Jessie from that environment. For now
there is nothing an individual Tool maintainer needs to do.
[0]: https://wikitech.wikimedia.org/wiki/News/Jessie_deprecation
Bryan - on behalf of the Cloud VPS admin team
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
Cross-posting Johan's message from wikitech-l to signal boost. This is
a long survey if you fill in all the questions, but it really does
help the Wikimedia Foundation and others select and prioritize work.
Bryan
---------- Forwarded message ---------
From: Johan Jönsson <jjonsson(a)wikimedia.org>
Date: Wed, Sep 11, 2019 at 11:15 AM
Subject: [Wikitech-l] The Community Insights survey (for volunteer devs)
To: Wikimedia developers <wikitech-l(a)lists.wikimedia.org>
Hey folks, I'm helping Rebecca Maung (rmaung(a)wikimedia.org) distribute
this request. Her words below:
The Wikimedia Foundation is asking for your feedback in the annual
Community Insights survey. We want to know how well we are supporting
your work on- and off-wiki, and how we can change or improve things in
the future. The opinions you share will directly affect the current
and future work of the Wikimedia Foundation.
If you are a volunteer developer, and have contributed code to any
pieces of MediaWiki, gadgets, or tools, please complete the survey. It
is available in various languages and will take between 15 and 25
minutes to complete.
Follow this link to the survey:
https://wikimedia.qualtrics.com/jfe/form/SV_0pSrrkJAKVRXPpj?Target=dev
If you have seen a similar message elsewhere and have already taken
the Community Insights survey, please do not take it twice.
You can find more information about this survey on the project page
and see how your feedback helps the Wikimedia Foundation support
contributors like you. This survey is hosted by a third-party service
and governed by this privacy statement. Please visit our frequently
asked questions page to find more information about this survey.
If you need additional help, send an email to surveys(a)wikimedia.org.
Thank you!
//Johan Jönsson
--
_______________________________________________
Wikitech-l mailing list
Wikitech-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808