October 2023 - Cloud - lists.wikimedia.org

Changes in schema of pagelinks tables
by Amir Sarabadani 15 Apr '24

15 Apr '24

(If you don’t work with pagelinks table, feel free to ignore this message) Hello, Here is an update and reminder on the previous announcement <https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/…> regarding normalization of links tables that was sent around a year ago. As part of that work, soon the pl_namespace and pl_title columns of pagelinks table will be dropped and you will need to use pl_target_id joining with the linktarget table instead. This is basically identical to the templatelinks normalization that happened a year ago. Currently, MediaWiki writes to both data schemes of pagelinks for new rows in all wikis except English Wikipedia and Wikimedia Commons (we will start writing to these two wikis next week). We have started to backfill the data with the new schema but it will take weeks to finish in large wikis. So if you query this table directly or your tools do, You will need to update them accordingly. I will write a reminder before dropping the old columns once the data has been fully backfilled. You can keep track of the general long-term work in T300222 <https://phabricator.wikimedia.org/T300222> and the specific work for pagelinks in T299947 <https://phabricator.wikimedia.org/T299947>. You can also read more on the reasoning in T222224 <https://phabricator.wikimedia.org/T222224> or the previous announcement <https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/…> . Thank you, -- *Amir Sarabadani (he/him)* Staff Database Architect Wikimedia Foundation <https://wikimediafoundation.org/>

3 5

Reminder about the externallinks migration
by Amir Sarabadani 13 Dec '23

13 Dec '23

Hello, We have communicated <https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/…> this change in August 2022 but here is a reminder that if you query externallinks table in wikireplicas, you will need to rework your queries. Changes are: el_to, el_index and el_index_60 fields will be dropped and you need to query el_to_domain_index and el_to_path fields instead. The data has been migrated in all wikis except English Wikipedia, Wikimedia Commons and Wikidata, you can keep track of the data migration in T326314 <https://phabricator.wikimedia.org/T326314>. In late June, we will stop updating the old fields and will start dropping them This is done to optimize storage of external links (drastically reducing its size by removing duplication) and enabling user requested features such as resolving two <https://phabricator.wikimedia.org/T14810> fifteen-year-old tickets <https://phabricator.wikimedia.org/T17218> or easier aggregating of external links domain [1], something that wasn’t possible until now. [1] For example, If you want to get list of the top-linked websites, you can query `SELECT el_to_domain_index, count(*) from externallinks group by el_to_domain_index order by count(*) desc limit 50;` Thank you and sorry for the inconvenience. -- *Amir Sarabadani (he/him)* Staff Database Architect Wikimedia Foundation <https://wikimediafoundation.org/>

3 4

Sign up for the first language community meeting on November 17th, 16:00 UTC!
by Srishti Sethi 31 Oct '23

31 Oct '23

Hello all, The first language community meeting is coming up in a few weeks - November 17th, 16:00 UTC. If you're interested, you can sign up on this wiki page: < https://www.mediawiki.org/w/index.php?title=Wikimedia_Language_engineering/…> [1]. We would like to invite all of you who are involved in creating content or managing technical aspects across different language communities. This will be a participant-driven meeting, where we collectively discuss specific technical issues related to our language wikis and work together to find possible solutions. This could involve anything from fixing a broken template on the Kurdish wiki to brainstorming ideas for growing content on the Tulu Wiktionary, currently in the Wikimedia Incubator, or celebrating the creation of Fon Wikipedia, to using MinT for content translation. You can view the meeting structure and add ideas for discussion to the notes document here: < https://etherpad.wikimedia.org/p/language-community-meeting-november-2023> [2] If you need interpretation support from English to another language, please let us know by sending an email to ssethi(a)wikimedia.org. Looking forward to your participation! Cheers, Srishti & Jon [1] https://www.mediawiki.org/w/index.php?title=Wikimedia_Language_engineering/… [2] https://etherpad.wikimedia.org/p/language-community-meeting-november-2023 *Srishti Sethi* Senior Developer Advocate Wikimedia Foundation <https://wikimediafoundation.org/>

1 0

[Cloud-announce] 2023-10-25 8:00 UTC: Planned Toolforge build service maintenance outage
by Slavina Stefanova 25 Oct '23

25 Oct '23

Hi, Toolforge's Harbor instance will briefly be down for a version upgrade from 2.5 to 2.9 this Wednesday at 8:00 UTC. https://phabricator.wikimedia.org/T346241 <https://phabricator.wikimedia.org/T346241> This should not affect any tools that are not using the new build service, nor any tools that are already running. https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service If you are using the builds service, you will not be able to run any new builds, or start a job or a webservice from an image built with the build service while Harbor is down. We will send an update before starting maintenance work, and once everything is back up and running. -- Slavina Stefanova (she/her) Software Engineer | Developer Experience Wikimedia Foundation _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 2

[Cloud-announce] Announcing Toolforge Build Service Open Beta
by Seyram Komla Sapaty 24 Oct '23

24 Oct '23

Hello, The Toolforge admin team is happy to announce that the Toolforge Build Service[0] is now available in open beta. The Build Service is intended to allow more tools to migrate off the Grid Engine and to make the process for deploying code to Toolforge easier and more flexible, by building container images with the specific dependencies for each tool. Here are quick highlights of some of the current key features: 1. Build your tool from source code, using you language's dependency management tool, no dockerfiles, no scripts, no manual steps 2. Use industry-wide standards[1] no vendor lock-in by using upstream buildpacks 3. Support for many languages out of the box[2] 4. Envvars - Create and manage environment variables and secrets that are available at runtime.[3] 5. Ability to install packages from the Ubuntu repositories[4] 6. Improved resiliency and resource usage by allowing NFS-less webservices[5], if you don't need NFS 7. Test your image locally, or anywhere[6] Please review the current known limitations here[7] We also have a growing list of tutorials for various languages[8] During this open beta, we invite you to actively participate and share your feedback replying to this thread or through irc, and if you find any issues or have any feature suggestions, you can use this task template[9]. Your insights will help us enhance and tailor the Build Service to meet the needs of your tools. The plan is to have this phase run for the next months, and if no big issues are found, promote it to global availability phase 1 (GA1) while we work on adding automatic triggering and deployment, for which we will do a second round of beta testing for those specific features. This unblocks the last step to migrate out of the grid, so we request all grid users to give it a try and report any issues they might find, there's no big changes expected for the currently implemented features, so any work done now will help later. Thank you for being a part of this journey. We look forward to your invaluable feedback and collaboration as we strive to provide a better developer experience. [0]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service [1]: https://buildpacks.io/ [2]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service#Supported_… [3]:https://wikitech.wikimedia.org/wiki/Help:Toolforge/Envvars_Service [4]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service#Installing… [5]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service#Using_NFS_… [6]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service#Testing_lo… [7]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service#Known_curr… [8]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service#Tutorials_… [9]: https://w.wiki/7kpi -- Seyram Komla Sapaty Developer Advocate Wikimedia Cloud Services _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

3 3

[Cloud-announce] 2023-10-23 13:00 UTC: Small outage planned for Openstack network cleanup
by David Caro 23 Oct '23

23 Oct '23

Hi! There will be a small network interruption next Monday at around 13:00 UTC as we will be doing some cleanup on the Openstack after network re-architecture (see https://phabricator.wikimedia.org/T348140). It will affect all the CloudVPS and other services hosted there (including toolforge, PAWS, quarry and superset). VMs traffic to the internet will be cut for a short period, hopefully for a few seconds, while internal traffic will not be affected, but if you have any open ssh session to your VMs or login.toolforge.org, it might timeout or get dropped, and any web access to projects will not work during the downtime. We will update on this email thread when we start and when we have finished. This will help stabilize the network and avoid bigger outages in the future, so thanks for your patience! -- David Caro SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3 "Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment." _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 2

First edition of the Language & Internationalization newsletter
by Srishti Sethi 16 Oct '23

16 Oct '23

Hello everyone, We are thrilled to introduce the first edition of the Language & Internationalization newsletter, available at this link: < https://www.mediawiki.org/wiki/Wikimedia_Language_engineering/Newsletter/20…> [1]. This newsletter is compiled by the Wikimedia Language team. It provides updates from July–September quarter on new feature development, improvements in various language-related technical projects and support efforts, details about community meetings, and contributions ideas to get involved in projects. To stay updated, you can subscribe to the newsletter on its wiki page. If you have any feedback or ideas for topics to feature in the newsletter, please share them on the discussion page, accessible here: < https://www.mediawiki.org/w/index.php?title=Talk:Wikimedia_Language_enginee…> [2]. Cheers, Srishti [1] https://www.mediawiki.org/wiki/Wikimedia_Language_engineering/Newsletter/20… [2] https://www.mediawiki.org/w/index.php?title=Talk:Wikimedia_Language_enginee… *Srishti Sethi* Senior Developer Advocate Wikimedia Foundation <https://wikimediafoundation.org/>

1 0

[Cloud-announce] Object storage (swift and s3) now available for some projects on Cloud VPS
by Andrew Bogott 13 Oct '23

13 Oct '23

CloudVPSnow offersobject storage, using both the S3 and the OpenStack Swift APIs. Horizon includes an 'Object store' tab which supports creation of containers and uploading of files. Some basic user documentation can be found on Wikitech[0]. For detailed API documentation, please consult the upstream documentation[1][2]. What doesn't work: - Due to '-' not being a valid character in Ceph tenant names, object storage will not work at all for projects with a - in their name. Feel free to open a a project request[3] for a new project with a simpler name, either to replace your project or to use alongside your current project for object storage. - Large files (greater than about 200M) cannot be uploaded from the Horizon UI. Large file support works fine via the openstack CLI using application credentials[4]. - Integrating this service with Toolforge remains as an open question. Currently only full cloud-vps project members will have access to object storage. If you run into other surprising or bad behaviors, please log tickets under https://phabricator.wikimedia.org/T276961 -Andrew + the WMCS team [0] https://wikitech.wikimedia.org/wiki/Help:Object_storage_user_guide [1] https://docs.ceph.com/en/quincy/radosgw/s3/#radosgw-s3 [2] https://docs.ceph.com/en/quincy/radosgw/swift/#radosgw-swift [3] https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS_project#Request_a_new_Cl… [4] https://wikitech.wikimedia.org/wiki/Help:Using_OpenStack_APIs#Application_C… _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 0

[Cloud-announce] Toolforge Kubernetes maintenance on 2023-10-18
by Taavi Väänänen 10 Oct '23

10 Oct '23

Hi, We will be upgrading the Toolforge Kubernetes cluster cluster on October 18th starting at around 11:00 UTC. The expected impact is that tools running on the Kubernetes cluster will get restarted a couple of times over the course of the few hours it takes for us to upgrade the entire cluster. This includes tools that use the jobs framework and tools that run web services using the default Kubernetes backend. The ability to manage tools will remain operational. Taavi -- Taavi Väänänen (he/him) Site Reliability Engineer, Cloud Services Wikimedia Foundation _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 0

Webservice logs?
by Roy Smith 06 Oct '23

06 Oct '23

I see T336057 was just closed. Looking at the docs <https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#View_web_service_logs>, I'm unclear how this works. The docs say ", the output from the webservice command is stored by the Toolforge Kubernetes infrastructure as long as the web service is running." So, what happens when a service exits (i.e. crashes)? Does that mean the logs for that service disappear?

2 2

2024

2023

2022

2021

2020

2019

2018

2017

Cloud October 2023