- Cloud - lists.wikimedia.org

Wednesday: Technical Advice IRC Meeting
by Michael Schönitzer 11 Mar '19

11 Mar '19

Reminder: Technical Advice IRC meeting this week **(Wednesday) 4-5 pm UTC** on #wikimedia-tech. Question can be asked in English, German & Spanish! The Technical Advice IRC Meeting is a weekly support event for volunteer developers. Every Wednesday, two full-time developers are available to help you with all your questions about Mediawiki, gadgets, tools and more! This can be anything from "how to get started" over "who would be the best contact for X" to specific questions on your project. If you know already what you would like to discuss or ask, please add your topic to the next meeting: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting Hope to see you there! Michi (for the Technical Advice IRC Meeting crew) -- Michael F. Schönitzer Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin Tel. (030) 219 158 26-0 https://wikimedia.de Unsere Vision ist eine Welt, in der alle Menschen am Wissens der Menschheit teilhaben, es nutzen und mehren können. Helfen Sie uns dabei! https://spenden.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

1 0

Fwd: [Cloud VPS alert] Puppet failure on tools-sgeexec-0905.tools.eqiad.wmflabs
by Martin Urbanec 11 Mar '19

11 Mar '19

I'm not sure I should be the intended recipient of this email.What happened there? Best, Martin ---------- Forwarded message --------- From: <root(a)tools-sgeexec-0905.tools.eqiad.wmflabs> Date: Mon, 11 Mar 2019 at 09:40 Subject: [Cloud VPS alert] Puppet failure on tools-sgeexec-0905.tools.eqiad.wmflabs To: <martin.urbanec(a)wikimedia.cz> Puppet is failing to run on the "tools-sgeexec-0905.tools.eqiad.wmflabs" instance in Wikimedia Cloud VPS. Working Puppet runs are needed to maintain instance security and logins. As long as Puppet continues to fail, this system is in danger of becoming unreachable. You are receiving this email because you are listed as member for the project that contains this instance. Please take steps to repair this instance or contact a Cloud VPS admin for assistance. For further support, visit #wikimedia-cloud on freenode or <https://wikitech.wikimedia.org>

3 2

[Cloud-announce] Toolforge: Bastion changes and Trusty deprecation final steps
by Bryan Davis 08 Mar '19

08 Mar '19

As announced previously on this list [0] we are in the process of replacing the old Ubuntu Trusty instances in Toolforge with fancy new Debian Stretch instances. == Remaining timeline == * Week of 2019-03-04: Switch login.tools.wmflabs.org to point to Stretch bastion * Week of 2019-03-25: Shutdown Trusty grid The DNS entry for "login.tools.wmflabs.org" will be updated to point to a Debian Stretch bastion rather than the old Ubuntu Trusty bastion soon (like right after I send this email). This change will cause many ssh clients to alert about a change in the ssh host fingerprint. Updated fingerprints will be posted on wikitech [1][2] once the switch has been made. The legacy Ubuntu Trusty bastion will still be reachable as "login-trusty.tools.wmflabs.org" until that instance is deleted during the week of 2019-03-25. In just over 2 weeks we will be shutting down the Trusty grid for good. Any tools that have not migrated to either the Stretch grid or the Kubernetes cluster at that point will be forcibly shutdown. Nothing will be deleted in the tools' $HOME directories, but any Trusty grid jobs will be stopped. Any crontab file remaining on the old grid's cron server will be archived as "$HOME/crontab.trusty.save". Maintainers who somehow missed all of the announcements will be able to login and restart their tools on the Stretch grid or Kubernetes. See <https://wikitech.wikimedia.org/wiki/News/Toolforge_Trusty_deprecation> for additional information and tips on common problems that have been found thus far. [0]: https://lists.wikimedia.org/pipermail/cloud-announce/2019-January/000122.ht… [1]: https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/login.tools.wmfla… [2]: https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/tools-dev.wmflabs… Bryan, on behalf of the Toolforge admin team -- Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org> [[m:User:BDavis_(WMF)]] Manager, Technical Engagement Boise, ID USA irc: bd808 v:415.839.6885 x6855 _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

1 0

[Cloud-announce] DSA (ssh-dss) SSH keys deprecated for Cloud VPS and Toolforge users
by Bryan Davis 02 Mar '19

02 Mar '19

Openssh 7.0, released 2015-08-11, deprecated the use of DSA (ssh-dss) keys and RSA keys smaller than 1024 bits [0]. We have been applying some backwards compatibility configuration changes to ssh bastion servers in both Cloud VPS and Toolforge for some time to continue to support old keys using these deprecated algorithms. I was supposed to announce this to the community about 1.5 years ago, but apparently I did not [1]. We have noticed with the introduction of Debian Stretch ssh bastion servers running Openssh 7.4 that users with DSA keys (and possibly short RSA keys) are being denied access by the newer software. The easiest fix for this is for users to generate new keys and upload their new public key using the form at <https://toolsadmin.wikimedia.org/profile/settings/ssh-keys> or <https://wikitech.wikimedia.org/wiki/Special:Preferences#mw-prefsection-open…>. We currently recommend using either ed25519 or 4096-bit RSA keys. See <https://wikitech.wikimedia.org/wiki/Production_shell_access#Generating_your…> for more information. [0]: https://www.openssh.com/txt/release-7.0 [1]: https://phabricator.wikimedia.org/T168433 Bryan, on behalf of the Wikimedia Cloud Services team -- Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org> [[m:User:BDavis_(WMF)]] Manager, Technical Engagement Boise, ID USA irc: bd808 v:415.839.6885 x6855 _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

1 0

[Cloud-announce] [Toolforge] max_user_connections = 20 set for ToolsDB
by Bryan Davis 25 Feb '19

25 Feb '19

During the flurry of activity we had recently in diagnosing and fixing problems with the shared ToolsDB MariaDB service [0], we made a configuration change to place a hard limit on the maximum number of simultaneous connections permitted for each user account [1][2]. The current limit is set at 20 concurrent connections. This should not cause any problems for a typical webservice or single script using ToolsDB, but tools making heavy use of ToolsDB may need to make some adjustments. As always, tool maintainers can seek advice on dealing with this limit or other issues in Toolforge from the Toolforge administration team and others in the community via our Freenode IRC channel (#wikimedia-cloud), Phabricator tasks, and the cloud(a)lists.wikimedia.org mailing list. [0]: https://phabricator.wikimedia.org/T216208 [1]: https://phabricator.wikimedia.org/T216170 [2]: https://mariadb.com/kb/en/library/server-system-variables/#max_user_connect… Bryan, on behalf of the Toolforge administration team -- Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org> [[m:User:BDavis_(WMF)]] Manager, Technical Engagement Boise, ID USA irc: bd808 v:415.839.6885 x6855 _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

1 0

Recovering from issues with NFS
by Giovanni Tirloni 25 Feb '19

25 Feb '19

Hello, Unfortunately, we had some disk space issues in our NFS server that required the NFS process to be restarted. This created all sorts of issues in Toolforge and Cloud VPS which we are still recovering from. If your tool is not working and wasn't restarted automatically, please try to restart it manually (`webservice restart`). We're truly sorry for the inconvenience. If you need help with a particular tool, please contact us on #wikimedia-cloud Regards, -- Giovanni Tirloni Operations Engineer Wikimedia Foundation

1 1

Wednesday: Technical Advice IRC Meeting
by Michael Schönitzer 20 Feb '19

20 Feb '19

Reminder: Technical Advice IRC meeting this week **(Wednesday) 4-5 pm UTC** on #wikimedia-tech. Question can be asked in English, Bulgarian & Hungarian. The Technical Advice IRC Meeting is a weekly support event for volunteer developers. Every Wednesday, two full-time developers are available to help you with all your questions about Mediawiki, gadgets, tools and more! This can be anything from "how to get started" over "who would be the best contact for X" to specific questions on your project. If you know already what you would like to discuss or ask, please add your topic to the next meeting: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting Hope to see you there! Michi (for the Technical Advice IRC Meeting crew) -- Michael F. Schönitzer Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin Tel. (030) 219 158 26-0 https://wikimedia.de Unsere Vision ist eine Welt, in der alle Menschen am Wissens der Menschheit teilhaben, es nutzen und mehren können. Helfen Sie uns dabei! https://spenden.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

1 0

[Cloud-announce] Toolsdb performance and outage issues
by Brooke Storm 19 Feb '19

19 Feb '19

This is an update on the ongoing problems with the toolsdb service. We are preparing to move to a new server, which is now a functioning replica of the toolsdb server. The first step here is to restart the service in read-only mode, and then we will move the DNS. Expect writes to stop working and connections to drop. When we are moved to the new DNS, services that use this database will need to be restarted. This will be happening within the next hour unless it is slowed down by some issues or caution. Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org> IRC: bstorm_ _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

6 6

Current status of Toolforge and Cloud VPS (2019-02-16)
by Arturo Borrero Gonzalez 16 Feb '19

16 Feb '19

Hi, Here is just a brief update on the status of Toolforge and CloudVPS by today 2019-02-16, along with some guess-estimations and what to expect in following days. Keeping track of all the events we had this week may be complex, because they were several of them, and heavily intermixed. * CloudVPS suffered severe hardware issues this week [0]. We solved most of the problems and added spare hardware [1] because our server capacity was really lowered. This service should be mostly stable right now. * Toolsdb (tools.db.svc.eqiad.wmflabs) is currently overloaded and suffering from hardware errors. We are already working on a replacement for this service [2]. Services depending on this database aren't working properly (like PAWS) and Toolforge tools that use it are also affected. An honest estimation is that services (specially Toolsdb) we won't be fully recovered until at least next Tuesday (2019-02-26). Our current plans involve replacing the Toolsdb hardware with virtual machines inside CloudVPS [3]. We are trying to be extra cautious to prevent data loss and other problems usually associated with doing things in a rush. Finally, I would like to mention that we are all well aware of the importance of these services for the community and we are doing our best to get things fixed. Thanks for your understanding and patience. regards [0] https://wikitech.wikimedia.org/wiki/Incident_documentation/20190213-cloudvps [1] CloudVPS: drain and rebuild labvirt1009 as cloudvirt1009 https://phabricator.wikimedia.org/T216239 [2] ToolsDB overload and cleanup https://phabricator.wikimedia.org/T216208 [3] Replace labsdb100[4567] with instances on cloudvirt1019 and cloudvirt1020 https://phabricator.wikimedia.org/T193264 -- Arturo Borrero Gonzalez Operations Engineer / Wikimedia Cloud Services Wikimedia Foundation

1 0

[Cloud-announce] Poor toolforge and PAWS performance for (at least) the next 24 hours
by Andrew Bogott 15 Feb '19

15 Feb '19

Because bad things come in threes (I'm hoping it's threes and not sevens) the server that hosts toolsdb is now also misbehaving. Brooke just now disabled a troubled drive which may have resolved things, but if the last few hours are any indication then the vast majority of connection or query attempts are likely to fail until we have a better solution in place. We're working on multiple fronts, trying to diagnose and fix the primary issue while also working to get new hardware online as a possible replacement server. Neither of those things are likely to get done until tomorrow, though, so toolforge will be in pretty bad shape in the meantime. It has been a rough couple of days, but rest assured we're taking notes about how to prevent outages like these in the future. Thank you for your patience in the meantime! -Andrew + the cloud team _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

3 2

2024

2023

2022

2021

2020

2019

2018

2017

Cloud