April 2020 - Cloud - lists.wikimedia.org

Toolforge: new domain toolforge.org
by Arturo Borrero Gonzalez 08 Jul '20

08 Jul '20

Hi! We are happy to announce the new domain 'toolforge.org' is now ready to be adopted by our Toolforge community. There is a lot of information related to this change in a wikitech page we have for this: https://wikitech.wikimedia.org/wiki/News/Toolforge.org The most important change you will see happening is a new domain/scheme for Toolforge-hosted webservices: * from https://tools.wmflabs.org/<toolname>/ * to https://<toolname>.toolforge.org/ A live example of this change can be found in our internal openstack-browser webservice tool: * legacy URL: https://tools.wmflabs.org/openstack-browser/ * new URL: https://openstack-browser.toolforge.org This domain change is something we have been working on for months previous to this announcement. Part of our work has been to ensure we have a smooth transition from the old domain (and URL scheme) to the new canonical one. However, we acknowledge the ride might be bumpy for some folks, due to technical challenges or cases we didn't consider when planning this migration. Please reach out intermediately if you find any limitation or failure anywhere related to this change. The wikitech page also contains a section with information for common problems. You can check now if your webservice needs any specific change by creating a temporal redirection to the new canonical URL: $ webservice --canonical --backend=kubernetes start [..] $ webservice --canonical --backend=gridengine start [..] The --canonical switch will create a temporal redirect that you can turn on/off. Please use this to check how your webservice behaves with the new domain/URL scheme. If you start the webservice without --canonical, the temporal redirect will be removed. We aim to introduce permanent redirects for the legacy URLs on 2020-06-15. We expect to keep serving legacy URLs forever, by means of redirections to the new URLs. More information on the redirections can also be found in the wikitech page. The toolforge.org domain is finally here! <3 -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

14 29

Loading wikipedia dump onto Clouds
by Huji Lee 05 May '20

05 May '20

Hi all, We have a set of database reports (on users, articles, etc.) that we used to generate on a weekly basis.[1] Ever since the introduction of the *actor* table,[2] many of the reports that have to do with users have become so slow that the SQL query cannot finish within a reasonable time and is killed. Some other reports have also become slower over time; all of these are shown in red in [1]. One possible solution is to create a script which is scheduled to run once a month; the script would download the latest dump of the wiki database,[3] load it into MySQL/MariaDB, create some additional indexes that would make our desired queries run faster, and generate the reports using this database. A separate script can then purge the data a few days later. We can use the current-version-only DB dumps for this purpose. I am guessing that this process would take several hours to run (somewhere between 2 and 10) and would require about 2 GB of storage just to download and decompress the dump file, and some additional space on the DB side (for data, indexes, etc.) Out of abundance of caution, I thought I should ask for permission now, rather than forgiveness later. Do we have a process for getting approval for projects that require gigabytes of storage and hours of computation, or is what I proposed not even remotely considered a "large" project, meaning I am being overly cautious? Please advise! Huji [1] https://fa.wikipedia.org/wiki/%D9%88%DB%8C%DA%A9%DB%8C%E2%80%8C%D9%BE%D8%AF… [2] https://phabricator.wikimedia.org/T223406 [3] https://dumps.wikimedia.org/fawiki/20200401/

4 8

Maintainers wanted: wikistream
by Bryan Davis 01 May '20

01 May '20

Wikistream is a tool that has been around for the Wikimedia movement for quite a while. It provides a web interface showing real-time editing data across many Wikimedia projects. The data for this comes from the IRC recent changes feeds. The webservice is written in nodejs and currently running on the Toolforge Kubernetes cluster. If you have experience using Toolforge and nodejs, see <https://phabricator.wikimedia.org/T251555> and apply to become a co-maintainer. Bryan -- Bryan Davis Technical Engagement Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808

1 0

Replica DB timeouts
by Federico Leva (Nemo) 30 Apr '20

30 Apr '20

[Again in the saga of me trying to query the revision and logging tables against comment text and usernames...] Am I dreaming or is the timeout on DB queries today something like 2 minutes? Is it a temporary measure? Is a query killer particularly aggressive due to some overload? Should we expect this to last? This query works: MariaDB [enwiki_p]> select count(*) from revision where rev_id > 950000000 AND rev_comment_id = 1334144; +----------+ | count(*) | +----------+ | 174 | +----------+ 1 row in set (1 min 57.35 sec) A slightly bigger one times out pretty quick: MariaDB [enwiki_p]> select count(*) from revision where rev_id > 930000000 AND rev_comment_id = 1334144; ERROR 2013 (HY000): Lost connection to MySQL server during query Federico

3 4

[Cloud-announce] Disk failure on cloudvirt1004 OpenStack host
by Bryan Davis 22 Apr '20

22 Apr '20

cloudvirt1004 is one of our oldest generation of hypervisor servers. The hypervisor servers are the machines which actually run the virtual machine instances for Cloud VPS projects. This physical host is experiencing an active hard disk and/or RAID controller failure. The Cloud Services team is actively attempting to fix the server and evacuate all instances running on it to other hypervisors. See <https://phabricator.wikimedia.org/T250869> for more information and progress updates. The following projects and instances are affected: * cloudvirt-canary ** canary1004-01.cloudvirt-canary.eqiad.wmflabs * commonsarchive ** commonsarchive-mwtest.commonsarchive.eqiad.wmflabs * deployment-prep ** deployment-echostore01.deployment-prep.eqiad.wmflabs ** deployment-schema-2.deployment-prep.eqiad.wmflabs * incubator ** incubator-mw.incubator.eqiad.wmflabs * machine-vision ** visionoid.machine-vision.eqiad.wmflabs * ogvjs-integration ** media-streaming.ogvjs-integration.eqiad.wmflabs * services ** Esther-outreachy-intern.services.eqiad.wmflabs * shiny-r ** discovery-testing-02.shiny-r.eqiad.wmflabs * tools ** tools-k8s-worker-38.tools.eqiad.wmflabs ** tools-k8s-worker-52.tools.eqiad.wmflabs ** tools-sgeexec-0901.tools.eqiad.wmflabs ** tools-sgewebgrid-lighttpd-0918.tools.eqiad.wmflabs ** tools-sgewebgrid-lighttpd-0919.tools.eqiad.wmflabs * toolsbeta ** toolsbeta-sgewebgrid-generic-0901.toolsbeta.eqiad.wmflabs * wikidata-autodesc ** wikidata-autodesc.wikidata-autodesc.eqiad.wmflabs * wikilink ** wikilink-prod.wikilink.eqiad.wmflabs Bryan, on behalf of the Cloud VPS admins and Cloud Services team -- Bryan Davis Technical Engagement Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808 _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

1 0

Changes to CloudVPS web proxy (XFF) on 2020-04-15
by Arturo Borrero Gonzalez 17 Apr '20

17 Apr '20

Hi there! If you use a CloudVPS web proxy, this email is for you. Toolforge developers/users can ignore this email. We are introducing a change to eliminate the 'X-Forwarded-For' HTTP header that the CloudVPS web proxy adds when forwarding the HTTP request to your instance. This header contains the original IP address of the internet client that sent the request. This is private information that we would like to reduce in our environment [0]. You use the web proxy if you have a public web endpoint hosted in CloudVPS under the wmflabs.org domain. These are generally configured using Horizon in the DNS > Web Proxies section. Examples of web proxy names: * accounts.wmflabs.org * glampipe.wmflabs.org * incubator.wmflabs.org Full list can be seen in the Openstack Browser tool [1]. We are ready to introduce this change [2], but wanted to give some heads up for projects that do require this information for whatever reason. We would like to hear from you in the next couple of weeks. Please contact us in the phabricator task [0] and include some rationale why you need the XFF header. This is the timeline this change will follow: * 2020-04-01: this email, start collecting list of things that require XFF * 2020-04-07: start evaluating list of things that require XFF * 2020-04-15: introduce the change, with proper case whitelisting When the change is introduced, in two weeks from now, proxy backends that were not whitelisted will stop receiving the XFF header. Please reach out for any questions or comments. regards. [0] https://phabricator.wikimedia.org/T135046 [1] https://openstack-browser.toolforge.org/project/project-proxy [2] https://gerrit.wikimedia.org/r/c/operations/puppet/+/583098 -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

4 5

[Cloud-announce] cloud-vps maintenance tomorrow, 2019-04-16
by Andrew Bogott 17 Apr '20

17 Apr '20

We'll be upgrading the cloud services OpenStack install tomorrow, beginning at 15:00 UTC. There should be little to no interruption to VMs or Toolforge, but Horizon logins will be disabled for part of the window. Sorry for the short notice! - Andrew + the WMCS team _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

1 2

Logging query with actors and log type (user stats)
by Maciej Jaros 16 Apr '20

16 Apr '20

Not sure if this is an appropriate place to discuss this but I'm not able to run this query: https://quarry.wmflabs.org/query/24267 Even a minor part of this query like below needs a significant time to complete. SELECT count(*) as review_count, log_actor FROM logging WHERE log_type = 'review' AND log_action = 'approve' GROUP BY log_actor I tried with logging_userindex but that don't seem to help. Any chances on adding extra indexes to make such queries work? I think querying log_type and log_actor should be quite common for various user stats. Even this takes almost 200 seconds to complete: SELECT count(*) as review_count, log_actor FROM logging_userindex WHERE log_type = 'review' AND log_action = 'approve' AND log_timestamp >= 20190101000000 AND log_timestamp <= 20190101235959 GROUP BY log_actor Also note that the query 24267 used to work in January 2020. I searched through the list, but haven't found any changes related to the logging table since January. Cheers, Nux.

2 3

[Mediawiki-api-announce] Change to action=login response when login fails due to session loss
by Brad Jorsch (Anomie) 15 Apr '20

15 Apr '20

Since April 2010,[1] when no lgtoken is passed to the Action API action=login it will return a "NeedToken" response including the token to use. While this method of fetching the login token was deprecated in January 2016,[2] it is still present for the benefit of clients that have not yet been updated and is not (yet) being removed. The NeedToken response was also being returned when an lgtoken was supplied but could not be validated due to session loss. While this made sense back in 2010 when the NeedToken response was the only way to fetch the login token, these days it is mainly confusing[3] and a way for clients with broken cookie handling to wind up in a loop. With the merge of Gerrit change 586448,[4] the API will no longer return NeedToken when lgtoken was supplied. If the token cannot be validated due to session loss, a "Failed" response will be returned with a message referring to session loss as the problem. This change should be deployed to Wikimedia sites with 1.35.0-wmf.28 or later, see https://www.mediawiki.org/wiki/MediaWiki_1.35/Roadmap for a schedule. Note that the change HAS NOT been deployed to Wikimedia sites as of the time of this email. If your client's ability to log in broke on 6 April 2020, the cause is most likely an unrelated change to Wikimedia's infrastructure that caused some HTTP headers to be output with HTTP/2 standard casing, i.e. "set-cookie" rather than the traditional "Set-Cookie". See https://phabricator.wikimedia.org/T249680 for details and further discussion of that situation. [1]: https://www.mediawiki.org/wiki/Special:Code/MediaWiki/64677 [2]: https://lists.wikimedia.org/pipermail/mediawiki-api-announce/2016-January/0… [3]: https://phabricator.wikimedia.org/T249526 [4]: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/586448 -- Brad Jorsch (Anomie) Senior Software Engineer Wikimedia Foundation _______________________________________________ Mediawiki-api-announce mailing list Mediawiki-api-announce(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce

2 1

Help smaller wiki communities grow their technical capacity!
by Srishti Sethi 14 Apr '20

14 Apr '20

Hello all, Small Wiki Toolkits (SWT) is an initiative to support small wiki communities by sharing and developing technical skills needed to support, maintain, and grow a language wiki. You can learn more about it here: https://meta.wikimedia.org/wiki/Small_wiki_toolkits As a next step for this initiative, we would like to share with you some ideas on how you can be a part of this: - Contribute to a Starter kit <https://phabricator.wikimedia.org/T242545> by proposing or helping develop and write recommendations for technical topics in simple English. - Develop toolkits using the recommendations <https://meta.wikimedia.org/wiki/Small_wiki_toolkits#Develop_toolkits> on the SWT page. You can see existing toolkits in the Category:Small_wiki_toolkits <https://commons.wikimedia.org/wiki/Category:Small_wiki_toolkits> on Wikimedia Commons. - Learn technical topics using the resources <https://meta.wikimedia.org/wiki/Small_wiki_toolkits#Toolkits> on the SWT page and apply the skills gained to provide technical support to a small wiki community. - Translate resources such as the SWT page itself and tutorials on technical topics linked from it. - Provide technical support <https://meta.wikimedia.org/wiki/Small_wiki_toolkits#Ask_your_technical_ques…> on the support desk on mediawiki.org, mailing lists, etc. - Organize a technical workshop or a session to help a small wiki community. If you want to help with any of the above or there is anything else you would like to share around building local capacity in a small wiki, please start a discussion on the talk page <https://meta.wikimedia.org/wiki/Talk:Small_wiki_toolkits> and/or add your name to the list of interested members <https://meta.wikimedia.org/wiki/Small_wiki_toolkits#Members_list>. Cheers, The Small Wiki Toolkits Initiative *Srishti Sethi* Developer Advocate Wikimedia Foundation <https://wikimediafoundation.org/>

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

Cloud April 2020