June 2022 - Cloud - lists.wikimedia.org

[Cloud-announce] Eliminating Debian Stretch in Cloud VPS
by Andrew Bogott 13 Oct '22

13 Oct '22

Debian Stretch's security support ends in mid 2022, and the Foundation's OS policy already discourages use of existing Stretch machines. That means that it's time for all project admins to start rebuilding your VMs with Bullseye (or, if you must, Buster.) Any webservices running in Kubernetes created in the last year or two are most likely using Buster images already, so there's no action needed for those. Older kubernetes jobs should be refreshed to use more modern images whenever possible. If you are still using the grid engine for webservices, we strongly encourage you to migrate your jobs to Kubernetes. For other grid uses, watch this space for future announcements about grid engine migration; we don't yet have a solution prepared for that. Details about the what and why for this process can be found here: https://wikitech.wikimedia.org/wiki/News/Stretch_deprecation Here is the deprecation timeline: March 2021: Stretch VM creation disabled in most projects July 6, 2021: Active support of Stretch ends, Stretch moves into LTS <- You are Here -> January 1st, 2022: Stretch VM creation disabled in all projects, deprecation nagging begins in earnest. Stretch alternatives will be available for tool migration in Toolforge May 1, 2022: All active Stretch VMs will be shut down (but not deleted) by WMCS admins. This includes Toolforge grid exec nodes. June 30, 2022: LTS support for Debian Stretch ends, all Stretch VMs will be deleted by WMCS admins _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 1

Changes in schema of MediaWiki links tables
by Amir Sarabadani 12 Aug '22

12 Aug '22

(If you don’t work with links tables such as templatelinks, pagelinks and so on, feel free to ignore this message) TLDR: The schema of links tables (starting with templatelinks) will change to have numeric id pointing to linktarget table instead of repeating namespace and title. Hello, The current schema and storage of most links tables are: page id (the source), namespace id of the target link and title of the target. For example, if a page with id of 1 uses Template:Foo, the row in the database would be 1, 6, and Foo (Template namespace has id of 6) Repeating the target’s title is not sustainable, for example more than half of Wikimedia Commons database is just three links tables. The sheer size of these tables makes a considerable portion of all queries slower, backups and dumps taking longer and taking much more space than needed due to unnecessary duplication. In Wikimedia Commons, on average a title is duplicated around 100 times for templatelinks and around 20 times for pagelinks. The numbers for other wikis depend on the usage patterns. Moving forward, these tables will be normalized, meaning a typical row will hold mapping of page id to linktarget id instead. Linktarget is a new table deployed in production and contains immutable records of namespace id and string. The major differences between page and linktarget tables are: 1- linktarget values won’t change (unlike page records that change with page move) 2- linktarget values can point to non-existent pages (=red links). The first table being done is templatelinks, then pagelinks, imagelinks and categorylinks will follow. During the migration phase both values will be accessible but we will turn off writing to the old columns once the values are backfilled and switched to be read from the new schema. We will announce any major changes beforehand but this is to let you know these changes are coming. While the normalization of all links tables will take several years to finish, templatelinks will finish in the next few months and is the most pressing one. So if you: - … rely on the schema of these tables in cloud replicas, you will need to change your tools. - … rely on dumps of these tables, you will need to change your scripts. Currently, templatelinks writes to both data schemes for new rows in most wikis. This week we will start backfilling the data with the new schema but it will take months to finish in large wikis. You can keep track of the general long-term work in https://phabricator.wikimedia.org/T300222 and the specific work for templatelinks in https://phabricator.wikimedia.org/T299417. You can also read more on the reasoning in https://phabricator.wikimedia.org/T222224. Thanks -- *Amir Sarabadani (he/him)* Staff Database Architect Wikimedia Foundation <https://wikimediafoundation.org/>

1 2

Password change on wikitech?
by Roy Smith 03 Jul '22

03 Jul '22

I couldn't log into wikitech or horizon this morning. A password reset fixed the problem. Did something happen to invalidate passwords?

2 1

Re: [Wikitech-l] Re: [Wikidata] [Small wiki toolkits] Upcoming bots & scripts workshop on Thursday, June 30th, 16:00 UTC
by Daniel Zahn 30 Jun '22

30 Jun '22

On Sat, Jun 18, 2022 at 2:54 AM Samuel Klein <meta.sj(a)gmail.com> wrote: > Is there any sort of list of mothballed bots looking for good homes / maintainers? I think basically every bot in the list would benefit from (more) maintainers: IRC bots: https://wikitech.wikimedia.org/wiki/Category:Bots on wiki bots: https://wikitech.wikimedia.org/wiki/Category:Bot_accounts And in general bots that work on Gerrit probably need their equivalent in Gitlab and bots working on IRC might need their equivalent on Slack (which unfortunately became standard for many but lack both volunteers and bots).

2 1

How to get django to serve robots.txt in toolforge?
by Roy Smith 20 Jun '22

20 Jun '22

I want to serve a robots.txt file from my django app (spt-tools). My settings.py file has: STATIC_ROOT = f'{WWW_DIR}/static/' so when I run "manage.py collectstatic", robots.txt ends up in $HOME/www/static/robots.txt. It needs to be one directory level up from there. I can think of several ways to fix this, but most of them involve adding additional steps to my install process, which I'd rather not have to do. I'm curious how other people have approached this.

2 1

[Small wiki toolkits] Upcoming bots & scripts workshop on Thursday, June 30th, 16:00 UTC
by Srishti Sethi 16 Jun '22

16 Jun '22

Hello everyone, The fifth workshop on the topic of "Hosting bots and scripts on Toolforge" is coming up - it will take place on Thursday, June 30th at 16:00 UTC. You can find more details on the workshop and a link to join here: < https://meta.wikimedia.org/wiki/Small_wiki_toolkits/Workshops#How_to_host_a…> [1]. This workshop will introduce participants to Toolforge, how to create a developer account, access it via ssh, create and run bots and scripts, etc. You can add your discussion ideas in the etherpad doc linked from the workshops page. To participate in this workshop, you would need basic familiarity with using command-line tools and technical knowledge of bots. We look forward to your participation! Best, Srishti On behalf of the SWT Workshops Organization team [1] https://meta.wikimedia.org/wiki/Small_wiki_toolkits/Workshops#How_to_host_a… *Srishti Sethi* Senior Developer Advocate Wikimedia Foundation <https://wikimediafoundation.org/>

1 0

Developer Portal is launched! Discover Wikimedia’s technical areas and how to contribute
by Birgit Müller 15 Jun '22

15 Jun '22

Hi All, We are happy to announce the launch of the Wikimedia Developer Portal <https://developer.wikimedia.org/> - a centralized entry point for finding technical documentation and community resources across Wikimedia’s key technical areas. The Developer Portal project <https://www.mediawiki.org/wiki/Developer_Advocacy/Developer_Portal> is part of a broader initiative to improve the discoverability and overall quality of key technical documentation. Work on this project began in July 2020 and included research and feedback rounds <https://www.mediawiki.org/wiki/Developer_Advocacy/Developer_Portal/History>, designing a content strategy <https://www.mediawiki.org/wiki/Developer_Advocacy/Developer_Portal/Content_…> and user journeys <https://www.mediawiki.org/wiki/Developer_Advocacy/Developer_Portal?tableofc…> to help people navigate and achieve their tasks, technical implementation <https://www.mediawiki.org/wiki/Developer_Advocacy/Developer_Portal/Implemen…> of the portal, developing a documentation review process <https://www.mediawiki.org/wiki/Documentation/Review_template>, and updating key documents. Wikimedia’s technical documentation spans a wide range of technologies, is distributed (and often duplicated) across mediawiki <https://www.mediawiki.org/wiki/MediaWiki>, wikitech <https://wikitech.wikimedia.org>, code repositories or other places, might or might not be up-to-date, and serves multiple audiences. This complex landscape can make it hard to find the information you need. The goal of the Developer Portal is to make it easier for developers and other technical contributors to: - Find the key documentation they need for common developer tasks. - Discover available tools and technologies. - Learn how to get started in Wikimedia technical areas. At its core, the Developer Portal is a navigation tool. It is an index of categorized links to key sources of technical information. These sources are hosted primarily on wikis—the portal itself contains no actual documentation. A major part of this project includes reviewing and updating the documents linked from the Developer Portal - the actual documentation (examples: Localisation <https://www.mediawiki.org/wiki/Localisation>, Communication <https://www.mediawiki.org/wiki/Communication>, Cloud Services introduction <https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_introduction>). This work will continue over the next year while also investigating how to improve and scale the process. The Developer Portal is a project by the Developer Advocacy team within the Technical Engagement group at the Wikimedia Foundation and has been developed by a project team of technical writers, engineers, and developer advocates (Alex <https://meta.wikimedia.org/wiki/User:APaskulin_(WMF)>, Andre <https://meta.wikimedia.org/wiki/User:AKlapper_(WMF)>, Bryan <https://meta.wikimedia.org/wiki/User:BDavis_(WMF)>, Sarah R <https://meta.wikimedia.org/wiki/User:SRodlund_(WMF)>, Tricia <https://meta.wikimedia.org/wiki/User:TBurmeister_(WMF)>). The long term goal is to step-by-step move towards a future where Wikimedia’s technical documentation is discoverable, accurate, standardized, and continuously updated. Yes, this is big :-) — and yes, it will take a while. Hope you join us in realizing this collective effort of making it easier for people to contribute to the code and technical spaces. Thank you <3 This project would not have been possible without the support, knowledge, ideas and feedback of many! A huge thanks to: - the members of WMF and WMDE engineering teams who participated in exploratory interviews early in the process, - everyone who provided feedback at Hackathon 2021 & 2022, via private messages, on the Project page <https://www.mediawiki.org/wiki/Developer_Advocacy/Developer_Portal/Content_…> or gave input or help on Phabricator tasks <https://phabricator.wikimedia.org/project/board/4941/query/all/> - Developers and community members who participated in multiple rounds of user testing <https://www.mediawiki.org/wiki/Developer_Advocacy/Developer_Portal/History#…> - the Design Strategy team <https://design.wikimedia.org/strategy/index.html> at the Foundation for helping us with designing a research study and recruiting a diverse group of users to test the final version of the site - people who helped with key doc improvements and content reviews: Haley <https://meta.wikimedia.org/wiki/User:HLepp_(WMF)>, Kamil <https://meta.wikimedia.org/wiki/User:KBach-WMF>, Komla <https://meta.wikimedia.org/wiki/User:SSapaty_(WMF)>, Nick <https://meta.wikimedia.org/wiki/User:Quiddity_(WMF)>, Srishti <https://meta.wikimedia.org/wiki/User:SSethi_(WMF)> - translators and translatewiki.net community, especially Abijeet <https://meta.wikimedia.org/wiki/User:APatro_(WMF)>, Niklas <https://meta.wikimedia.org/wiki/User:Nlaxstrom-WMF>, and Verdy_p <https://translatewiki.net/wiki/User:Verdy_p> - People who helped us with steps towards production and deployment: Alexandros <https://meta.wikimedia.org/wiki/User:AKosiaris_(WMF)>, Andrew <https://meta.wikimedia.org/wiki/User:AndrewBogott_(WMF)>, Brian <https://meta.wikimedia.org/wiki/User:BKing_(WMF)>, Ryan <https://meta.wikimedia.org/wiki/User:RKemper_(WMF)>, Valentin <https://meta.wikimedia.org/wiki/User:VGuti%C3%A9rrez_(WMF)> We hope that this new resource will be fun to explore and ultimately be useful for your work. If you are interested in learning more about the thinking behind the user journey design for the Developer Portal, please see the blog post on Diff <https://diff.wikimedia.org/2022/06/15/illuminating-pathways-to-technical-do…>. For more information about the technical implementation and design principles, please see the corresponding post on the techblog <https://techblog.wikimedia.org/2022/06/15/new-discovery-tool-for-technical-…>. All key information is also available on the project page <https://www.mediawiki.org/wiki/Developer_Advocacy/Developer_Portal> of the Developer Portal. Feedback, ideas and questions are more than welcome on the talk page of the project <https://www.mediawiki.org/wiki/Talk:Developer_Advocacy/Developer_Portal>. Let's keep discussions in one place. :-) Birgit & the Developer Portal project team -- Birgit Müller (she/her) Director of Technical Engagement Wikimedia Foundation <https://wikimediafoundation.org/>

2 1

[Cloud-announce] Rstudio in PAWS
by Vivian Rook 14 Jun '22

14 Jun '22

Rstudio has an interest in system users existing that make it difficult to upgrade in PAWS, which doesn't have the same ideas about system users. Currently the single user container is blocked from being upgraded by Rstudio. There are some more Rstudio specific details in https://phabricator.wikimedia.org/T302164 One simple solution would be to remove Rstudio. At the moment there is not a clear path forward for keeping it as a part of PAWS. Are there any opinions on this approach? Thank you -- *Vivian Rook (They/Them)* Site Reliability Engineer Wikimedia Foundation <https://wikimediafoundation.org/> _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 0

ECDSA host key
by John Shepherd 13 Jun '22

13 Jun '22

2 1

High replag on s1/enwiki wiki replicas
by Kunal Mehta 10 Jun '22

10 Jun '22

Hi, Because of schema changes happening in production, the s1/enwiki wiki replica is currently behind by ~40 hours, so tools and bots relying on it will have outdated data. The expectation is that it should recover by Monday. You can look up the current replag using <https://replag.toolforge.org/> and discussion about the issue itself is at <https://phabricator.wikimedia.org/T310325>. -- Legoktm

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Cloud June 2022