Cloud March 2019

cloud@lists.wikimedia.org

23 participants
20 discussions

[Cloud-announce] Dropping some comment-related columns in the wiki replicas
by Brooke Storm 19 Mar '19

19 Mar '19

Since the MCR refactor of the Mediawiki database schema has been progressing (https://phabricator.wikimedia.org/T166733 <https://phabricator.wikimedia.org/T166732> and many other tickets), one of the last steps is dropping the columns from the wiki replica schema. The column drops are being tracked and are explained well here https://phabricator.wikimedia.org/T212972 <https://phabricator.wikimedia.org/T212972>, and currently the change is ready to be applied. It is already applied to two wikis (eswiki and huwiki since there was a column problem that needed fixing). The tables with names such as <tablename>_compat will retain a similar structure if that is needed for refactoring. From the ticket, this is a summary of what is changing, organized by table name: archive: Remove ar_comment archive_userindex: Remove ar_comment filearchive: Remove fa_deleted_reason Remove fa_description filearchive_userindex: Remove fa_deleted_reason Remove fa_description image: Remove img_description ipblocks: Remove ipb_reason. ipblocks_ipindex: Remove ipb_reason. logging: Remove log_comment. logging_logindex: Remove log_comment. logging_userindex: Remove log_comment. oldimage: Remove oi_description oldimage_userindex: Remove oi_description recentchanges: Remove rc_comment. recentchanges_userindex: Remove rc_comment. revision: Remove rev_comment. revision_userindex: Remove rev_comment. The changes to the _compat tables should not affect anything. We will deploy the change early next week (Tuesday - 3/12/2019). In most cases, if a table isn’t working for your tool or app anymore because of the change, you can switch to a table named $tablename_compat and it will appear to have the same schema, but it is recommended that comment references use a join to the new comment table on a “comment_id” field instead where possible. Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org> IRC: bstorm_ _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

4 8

Phabricator down?
by Arthur Smith 18 Mar '19

18 Mar '19

Things seem broken today - anything anybody can point to regarding the problems? Arthur

4 5

[Cloud-announce] Disabling creation of new Jessie images
by Andrew Bogott 18 Mar '19

18 Mar '19

tl;dr: We're about to disable self-service creation of Debian Jessie VMs. To request an exception, open a Phabricator ticket specifying your need and reasons. -- We're close to polishing off the last few Ubuntu Trusty VMs in the cloud, which means it's time to start thinking about the upcoming deprecation of Debian Jessie. WMCS (and the WMF in general) will continue to support use of Jessie well into 2020, so no immediate action is needed on the part of current Jessie users. On the other hand, any /new/ work should definitely happen on Stretch in order to postpone the inevitable OS-motivated rebuilds as long as possible. In order to encourage that, we're going to disable creation of new Jessie VMs in the next few days If you believe that you are a special case and need a Jessie VM anyway, please open a phabricator ticket explaining your reasons and specifying name and flavor for the VM to be created, and WMCS staff will make it for you. For reference, the phabricator ticket about this change is: https://phabricator.wikimedia.org/T218119 -Andrew + the WMCS team _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

1 0

[Cloud-announce] Filesize limit on Toolforge
by Brooke Storm 14 Mar '19

14 Mar '19

Due to repeated recent outages in the past 30 days and a long history of previous outages due to log files filling up NFS for Toolforge, I’ve deployed a change to restrict the maximum file size that can be created from the Toolforge system of 50 GB. When a process hits that limit, it will fail to continue writing to the file with the message that the “maximum file size” has been reached. There are files over this size in the environment now that are not going to be affected, but moving them within the same filesystem is likely to require help from someone with root access. If the limit becomes a problem, it can be revisited. Please let us know on the cloud discussion list or on #wikimedia-cloud if problems arise from the change. Thanks! Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org> IRC: bstorm_ _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

3 2

[Cloud-announce] Rebooting the stretch grid cron server on toolforge
by Brooke Storm 13 Mar '19

13 Mar '19

This is just a heads up that due to some residual issues from NFS problems, we are rebooting the cron server for the newer Stretch gridengine on toolforge. This may affect a small number of job submissions, but it should only affect those that happen during the reboot itself. Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org> IRC: bstorm_ _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

1 0

Fwd: Cron <tools.urbanecmbot@tools-sgecron-01> /usr/bin/jsub -N afdNoticesZlobot -once -quiet python3 /data/project/urbanecmbot/11bots/cswiki/userbots/afdHelper/zlobot-afd-notices.py
by Martin Urbanec 11 Mar '19

11 Mar '19

---------- Forwarded message --------- From: Martin Urbanec <martin.urbanec(a)wikimedia.cz> Date: Mon, 11 Mar 2019 at 16:06 Subject: Fwd: Cron <tools.urbanecmbot@tools-sgecron-01> /usr/bin/jsub -N afdNoticesZlobot -once -quiet python3 /data/project/urbanecmbot/11bots/cswiki/userbots/afdHelper/zlobot-afd-notices.py To: <cloud-l(a)lists.wikimedia.org> Doesn't look anything I did wrong, but...what is this? Martin ---------- Forwarded message --------- From: Cron Daemon <root(a)tools.wmflabs.org> Date: Mon, 11 Mar 2019 at 16:06 Subject: Cron <tools.urbanecmbot@tools-sgecron-01> /usr/bin/jsub -N afdNoticesZlobot -once -quiet python3 /data/project/urbanecmbot/11bots/cswiki/userbots/afdHelper/zlobot-afd-notices.py To: <martin.urbanec(a)wikimedia.cz> error: failed receiving gdi request response for mid=1 (got syncron message receive timeout error). Traceback (most recent call last): File "/usr/bin/job", line 48, in <module> root = xml.etree.ElementTree.fromstring(proc.stdout.read()) File "/usr/lib/python3.5/xml/etree/ElementTree.py", line 1345, in XML return parser.close() xml.etree.ElementTree.ParseError: no element found: line 1, column 0

5 4

Wednesday: Technical Advice IRC Meeting
by Michael Schönitzer 11 Mar '19

11 Mar '19

Reminder: Technical Advice IRC meeting this week **(Wednesday) 4-5 pm UTC** on #wikimedia-tech. Question can be asked in English, German & Spanish! The Technical Advice IRC Meeting is a weekly support event for volunteer developers. Every Wednesday, two full-time developers are available to help you with all your questions about Mediawiki, gadgets, tools and more! This can be anything from "how to get started" over "who would be the best contact for X" to specific questions on your project. If you know already what you would like to discuss or ask, please add your topic to the next meeting: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting Hope to see you there! Michi (for the Technical Advice IRC Meeting crew) -- Michael F. Schönitzer Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin Tel. (030) 219 158 26-0 https://wikimedia.de Unsere Vision ist eine Welt, in der alle Menschen am Wissens der Menschheit teilhaben, es nutzen und mehren können. Helfen Sie uns dabei! https://spenden.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

1 0

Fwd: [Cloud VPS alert] Puppet failure on tools-sgeexec-0905.tools.eqiad.wmflabs
by Martin Urbanec 11 Mar '19

11 Mar '19

I'm not sure I should be the intended recipient of this email.What happened there? Best, Martin ---------- Forwarded message --------- From: <root(a)tools-sgeexec-0905.tools.eqiad.wmflabs> Date: Mon, 11 Mar 2019 at 09:40 Subject: [Cloud VPS alert] Puppet failure on tools-sgeexec-0905.tools.eqiad.wmflabs To: <martin.urbanec(a)wikimedia.cz> Puppet is failing to run on the "tools-sgeexec-0905.tools.eqiad.wmflabs" instance in Wikimedia Cloud VPS. Working Puppet runs are needed to maintain instance security and logins. As long as Puppet continues to fail, this system is in danger of becoming unreachable. You are receiving this email because you are listed as member for the project that contains this instance. Please take steps to repair this instance or contact a Cloud VPS admin for assistance. For further support, visit #wikimedia-cloud on freenode or <https://wikitech.wikimedia.org>

3 2

[Cloud-announce] Toolforge: Bastion changes and Trusty deprecation final steps
by Bryan Davis 08 Mar '19

08 Mar '19

As announced previously on this list [0] we are in the process of replacing the old Ubuntu Trusty instances in Toolforge with fancy new Debian Stretch instances. == Remaining timeline == * Week of 2019-03-04: Switch login.tools.wmflabs.org to point to Stretch bastion * Week of 2019-03-25: Shutdown Trusty grid The DNS entry for "login.tools.wmflabs.org" will be updated to point to a Debian Stretch bastion rather than the old Ubuntu Trusty bastion soon (like right after I send this email). This change will cause many ssh clients to alert about a change in the ssh host fingerprint. Updated fingerprints will be posted on wikitech [1][2] once the switch has been made. The legacy Ubuntu Trusty bastion will still be reachable as "login-trusty.tools.wmflabs.org" until that instance is deleted during the week of 2019-03-25. In just over 2 weeks we will be shutting down the Trusty grid for good. Any tools that have not migrated to either the Stretch grid or the Kubernetes cluster at that point will be forcibly shutdown. Nothing will be deleted in the tools' $HOME directories, but any Trusty grid jobs will be stopped. Any crontab file remaining on the old grid's cron server will be archived as "$HOME/crontab.trusty.save". Maintainers who somehow missed all of the announcements will be able to login and restart their tools on the Stretch grid or Kubernetes. See <https://wikitech.wikimedia.org/wiki/News/Toolforge_Trusty_deprecation> for additional information and tips on common problems that have been found thus far. [0]: https://lists.wikimedia.org/pipermail/cloud-announce/2019-January/000122.ht… [1]: https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/login.tools.wmfla… [2]: https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/tools-dev.wmflabs… Bryan, on behalf of the Toolforge admin team -- Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org> [[m:User:BDavis_(WMF)]] Manager, Technical Engagement Boise, ID USA irc: bd808 v:415.839.6885 x6855 _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

1 0

[Cloud-announce] DSA (ssh-dss) SSH keys deprecated for Cloud VPS and Toolforge users
by Bryan Davis 02 Mar '19

02 Mar '19

Openssh 7.0, released 2015-08-11, deprecated the use of DSA (ssh-dss) keys and RSA keys smaller than 1024 bits [0]. We have been applying some backwards compatibility configuration changes to ssh bastion servers in both Cloud VPS and Toolforge for some time to continue to support old keys using these deprecated algorithms. I was supposed to announce this to the community about 1.5 years ago, but apparently I did not [1]. We have noticed with the introduction of Debian Stretch ssh bastion servers running Openssh 7.4 that users with DSA keys (and possibly short RSA keys) are being denied access by the newer software. The easiest fix for this is for users to generate new keys and upload their new public key using the form at <https://toolsadmin.wikimedia.org/profile/settings/ssh-keys> or <https://wikitech.wikimedia.org/wiki/Special:Preferences#mw-prefsection-open…>. We currently recommend using either ed25519 or 4096-bit RSA keys. See <https://wikitech.wikimedia.org/wiki/Production_shell_access#Generating_your…> for more information. [0]: https://www.openssh.com/txt/release-7.0 [1]: https://phabricator.wikimedia.org/T168433 Bryan, on behalf of the Wikimedia Cloud Services team -- Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org> [[m:User:BDavis_(WMF)]] Manager, Technical Engagement Boise, ID USA irc: bd808 v:415.839.6885 x6855 _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

Cloud March 2019