- Cloud-announce - lists.wikimedia.org

Maps and Scratch NFS migration
by Brooke Storm 14 May '19

14 May '19

To move maps project/home NFS and the scratch share off of the old labstore1003 machine and onto much faster, newer hardware, I’m going to begin doing rsyncs of data across. This is just to announce this is starting soon and to encourage people to reach out on the #wikimedia-cloud channel if it is hitting performance hard on maps servers, in particular or on the scratch share. Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org> IRC: bstorm_

1 0

[Breaking change] Important for Wikidata tools maintainers: wb_terms table to be dropped at the end of May
by Bryan Davis 24 Apr '19

24 Apr '19

The 'wb_terms' table is being removed from the Wiki Replica databases. Please see Léa Lacroix's post on the wikidata mailing list [0] for additional details. TL;DR summary: * May-June 2019, the Wikidata development team will drop the wb_terms table from the database in favor of a new optimized schema * Migration will start on 2019-05-29 * A test system will be available starting 2019-05-15 * Details are available in Phabricator [1] [0]: https://lists.wikimedia.org/pipermail/wikidata/2019-April/012987.html [1]: https://phabricator.wikimedia.org/T221764 Bryan -- Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org> [[m:User:BDavis_(WMF)]] Manager, Technical Engagement Boise, ID USA irc: bd808 v:415.839.6885 x6855

1 0

Partial toolforge and PAWS outages Monday
by Andrew Bogott 16 Apr '19

16 Apr '19

Tuesday starting at around 17:00 UTC I'm going to relocate the paws and kubernetes masters to the new network region. While the VMs are copying, launches of new kubernetes jobs and creation of new PAWS notebooks will fail. The outage should last about an hour -- less if everything goes well, somewhat more if not. Jobs that are already running when the copy begins should be unaffected. Apologies for any inconvenience caused! -Andrew

1 4

Re: [Cloud-announce] [Wikitech-l] Change to Wikitech logins: Username now case-sensitive
by Andrew Bogott 16 Apr '19

16 Apr '19

On 4/16/19 7:59 AM, Andrew Otto wrote: > Great! Is this just for Wikitech itself or all ldap/wikitech > authentication? This notice is related to a change in mediawiki code, so concerns direct logins to wikitech itself. That said, the 2fa key used by Horizon is stored in a the wikitech database, so it's vaguely possible that Horizon logins could be disrupted as well. Other services that rely on ldap for account creation (e.g. gerrit, icinga, etc.) are unaffected, although they may have unrelated case-(in)sensitive issues of their own. > > On Mon, Apr 15, 2019 at 7:56 PM Bryan Davis <bd808(a)wikimedia.org> wrote: > >> A change was deployed to the Wikitech config 2019-04-15T23:16 UTC >> which prevents users from logging into the wiki with a username that >> differs in case from the 'cn' value for their developer account. >> >> This change is not expected to cause problems for most users, but >> there may be some people who have historically entered a username with >> mismatched case (for example "bryandavis" instead of "BryanDavis") and >> relied on MediaWiki and the LdapAuthentication plugin figuring things >> out. This will no longer happen automatically. These users will need >> to update their password managers (or brains if they are not using a >> password manager) to supply the username with correct casing. >> >> The "wrongpassword" error message on Wikitech has been updated with a >> local override to help people discover this problem. See >> <https://phabricator.wikimedia.org/T165795> for more details. >> >> Bryan, on behalf of the Cloud Services team >> -- >> Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org> >> [[m:User:BDavis_(WMF)]] Manager, Technical Engagement Boise, ID USA >> irc: bd808 v:415.839.6885 x6855 >> >> _______________________________________________ >> Wikitech-l mailing list >> Wikitech-l(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/wikitech-l > _______________________________________________ > Wikitech-l mailing list > Wikitech-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-l

1 0

Change to Wikitech logins: Username now case-sensitive
by Bryan Davis 15 Apr '19

15 Apr '19

A change was deployed to the Wikitech config 2019-04-15T23:16 UTC which prevents users from logging into the wiki with a username that differs in case from the 'cn' value for their developer account. This change is not expected to cause problems for most users, but there may be some people who have historically entered a username with mismatched case (for example "bryandavis" instead of "BryanDavis") and relied on MediaWiki and the LdapAuthentication plugin figuring things out. This will no longer happen automatically. These users will need to update their password managers (or brains if they are not using a password manager) to supply the username with correct casing. The "wrongpassword" error message on Wikitech has been updated with a local override to help people discover this problem. See <https://phabricator.wikimedia.org/T165795> for more details. Bryan, on behalf of the Cloud Services team -- Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org> [[m:User:BDavis_(WMF)]] Manager, Technical Engagement Boise, ID USA irc: bd808 v:415.839.6885 x6855

1 0

OSMDB migration to new server 20190404@1700 UTC
by Brooke Storm 04 Apr '19

04 Apr '19

The OSM postgresql database service, usually accessed via osmdb.eqiad.wmnet is moving to a new server. Currently the server is a read replica of the primary database, and should be accessible via the DNS alias of osm.db.svc.eqiad.wmflabs. As detailed here https://phabricator.wikimedia.org/T219652 <https://phabricator.wikimedia.org/T219652>, osmdb.eqiad.wmnet will be changed to point at the osm.db.svc.eqiad.wmflabs. For a brief time that will make those tables that aren’t always read-only also read-only while DNS updates. Then the replica will be promoted to the master, and the rest of the steps should not cause any impact. Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org> IRC: bstorm_

1 3

PAWS master reboot
by Brooke Storm 27 Mar '19

27 Mar '19

Because of some issues mounting NFS on the PAWS master, it is being rebooted. Traffic to the front page has already been routed through another node, but server creation won’t work until after the reboot is complete. Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org> IRC: bstorm_

1 1

[Toolforge] Trusty job grid has been shutdown
by Bryan Davis 25 Mar '19

25 Mar '19

The legacy Ubuntu Trusty grid engine job grid has been shutdown! Thanks to everyone who was involved in migrating existing tools from the old grid to the Kubernetes cluster or the new Debian Stretch job grid. There were still 385 tools that may have been running jobs or webservices on the Trusty grid at the time of shutdown. A static list of these tools is preserved at <https://tools.wmflabs.org/trusty-tools/>. Instructions are still available at <https://wikitech.wikimedia.org/wiki/News/Toolforge_Trusty_deprecation> for migrating tools that are currently down. Bryan, on behalf of the Toolforge admin team -- Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org> [[m:User:BDavis_(WMF)]] Manager, Technical Engagement Boise, ID USA irc: bd808 v:415.839.6885 x6855

1 0

[Toolforge] Ubuntu Trusty job grid will be shutdown Monday 2019-03-25
by Bryan Davis 25 Mar '19

25 Mar '19

As previously announced on this list [0][1] we are in the process of replacing the old Ubuntu Trusty instances in Toolforge with fancy new Debian Stretch instances. This process is reaching its next major milestone on Monday 2019-03-25. During the general US workday on that date (14:00-00:00 UTC) the Toolforge admin team will be dismantling the legacy Ubuntu Trusty job grid. Any tools that have not migrated to either the Stretch grid or the Kubernetes cluster at that point will be forcibly shutdown. Nothing will be deleted in the tools' $HOME directories, but any Trusty grid jobs will be stopped. Any crontab file remaining on the old grid's cron server will be archived as "$HOME/crontab.trusty.save". Maintainers who somehow missed all of the announcements will be able to login and restart their tools on the Stretch grid or Kubernetes. See <https://wikitech.wikimedia.org/wiki/News/Toolforge_Trusty_deprecation> for additional information and tips on common problems that have been found thus far. [0]: https://lists.wikimedia.org/pipermail/cloud-announce/2019-January/000122.ht… [1]: https://lists.wikimedia.org/pipermail/cloud-announce/2019-March/000142.html Bryan, on behalf of the Toolforge admin team -- Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org> [[m:User:BDavis_(WMF)]] Manager, Technical Engagement Boise, ID USA irc: bd808 v:415.839.6885 x6855

1 1

Horizon logins currently broken
by Andrew Bogott 22 Mar '19

22 Mar '19

Good morning! As a side-effect of our response to the current gerrit vandalism epidemic, the 2fa integration between Horizon and Wikitech has been disabled. That means that existing Horizon sessions are still valid but fresh logins will fail. This problem is being actively worked on. In the meantime, don't panic if you get an error while trying to log in. -Andrew

1 1

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-announce