- Cloud - lists.wikimedia.org

greeting
by everyone Рит 17 Jul '21

17 Jul '21

Hi everyone 🔝🔥💯🛡️✔️

1 0

[Cloud-announce] 2021-07-20@1500 UTC Maps and Scratch NFS briefly unavailable
by Brooke Storm 16 Jul '21

16 Jul '21

Network maintenance will be happening on Tuesday, July 20th at around 1500 UTC that will affect the maps and scratch cluster on both nodes (see https://phabricator.wikimedia.org/T286069 <https://phabricator.wikimedia.org/T286069>). It should be extremely short in duration (measured in seconds, not minutes). Therefore, we will not be failing them over. WMCS will keep an eye on the impact to client VMs and will remediate problems where necessary. If all goes well, most services won’t notice. Brooke Storm Staff SRE Wikimedia Cloud Services bstorm(a)wikimedia.org _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 0

Two open developer positions for OpenRefine (paid contractors, part time, remote)
by sandra.fauconnier＠gmail.com 08 Jul '21

08 Jul '21

Hi everyone, I'm very happy to announce: OpenRefine [1] has two Junior Developer job openings (paid contractor positions; part-time, fully remote) for building Structured Data on Wikimedia Commons [2] functionalities. Needless to say, we would love to receive applications from Wikimedians :-) * Junior Developer - Wikimedia Development [3] (6 months, from September 2021 till February 2022) * Junior Developer - OpenRefine Development [4] (8 months, from November 2021 till June 2022) All the best! Sandra (User:Spinster / User:SFauconnier) [1] https://openrefine.org [2] https://w.wiki/UR [3] https://openrefine.org/blog/2021/07/07/Wikimedia-Commons-reconciliation-bat… [4] https://openrefine.org/blog/2021/07/07/OpenRefine-SDC-developer.html

1 0

[Cloud-announce] Webservice release changes coming this week
by Brooke Storm 02 Jul '21

02 Jul '21

Hello cloud users, We will be releasing webservice version 0.75 this week to Toolforge. Most of the changes should not be noticed (upgrading to python 3, preparing for grid system upgrades in the future), however, there is a change for some people who have started to use the service.template feature (https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Webservice_templates <https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Webservice_templates>). This change https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/636… <https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/636…> affects the way webservice templates are found by adding more locations. It will automatically check certain code directories for your template besides $HOME, but it will throw an error if you have a symlink to one of these locations back to your tool’s home directory. The locations besides $HOME that this checks for a service.template are * ~/www/python/src * ~/www/js * ~/public_html If you are affected by that, then simply remove the symlink in $HOME, and it should work fine. If you are not symlinking a service.template back to your tool’s home directory, you should not notice any changes. The full changelog is at https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/700… <https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/700…> When the release is complete, I’ll send a followup email and record it in SAL. Brooke Storm Staff SRE Wikimedia Cloud Services bstorm(a)wikimedia.org _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 1

[Cloud-announce] 2021-07-01 scratch and maps NFS maintenance
by Brooke Storm 01 Jul '21

01 Jul '21

The NFS servers used for scratch and maps mounts (/data/project and /home in the maps project and /data/scratch in other projects) will be going offline for a short time tomorrow 2021-07-01 at around 1600 UTC to move the mounts to DRBD synced volumes. The current setup causes odd issues during failover including data loss and stale files left behind. The process taking place is one of those failovers so there may be some files that were previously deleted that need deleting again present and similar anomalies. I plan to reboot the maps project servers to make sure they have their mounts and processes restored as best as possible. The scratch mounts should be less impactful. If you use scratch, just be aware that it will go offline for a bit and will be back with some possible quirks. After that, the data should become far more stable and properly synced between the two systems. The process could start later than 1600 UTC if there are sync issues initially as I try to get as much of the data as possible transferred. More details here https://phabricator.wikimedia.org/T224747 <https://phabricator.wikimedia.org/T224747> Brooke Storm Staff SRE Wikimedia Cloud Services bstorm(a)wikimedia.org _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 2

[Cloud-announce] Toolforge kubernetes upgrade Tuesday, 2021-06-29 at 17:00 UTC
by Andrew Bogott 29 Jun '21

29 Jun '21

Next Tuesday we will be upgrading Kubernetes on toolforge. As part of the upgrade we will need to restart all pods. This will produce a brief interruption in web services and other tools that use kubernetes. Assuming your services are able to survive a restart, no action should be needed on your part. I'll send a further email when the upgrade is finished. Special thanks to volunteer Taavi (aka Majavah) who has been essential in preparing for this upgrade and will be taking time out of his day to make sure the upgrade goes smoothly on Tuesday. -Andrew + the WMCS team _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 2

Changes to storage of files metadata
by Amir Sarabadani 27 Jun '21

27 Jun '21

Hello, If you don't do anything with metadata fields of file tables (image table for example) in replicas, you can ignore this email. "image" table in Wikimedia Commons is extremely big (more than 380GB compressed) and has been causing multiple major issues (including an incident recently). Deep inspections revealed that more than 80% of this table is metadata of PDF files, around 10% is metadata of DjVu files and the 10% left is the rest of the information. This clearly needs fixing. The work has been done on this by Tim Starling and we are slowly rolling out two major changes: First, format of metadata in the database (for example img_metadata field in image table) will change for all files. It used to be php serialization but it will be changed to json. You can see an example of before and after in https://phabricator.wikimedia.org/T275268#7178983 Keep it in mind that for some time this will be a hybrid mode that some files will have it in json format and some will have it in php serialization. You need to support both formats for a while if you parse this value. Second, some parts of metadata for PDF and later DjVu files won't be accessible in Wikimedia Cloud anymore. Since these data will be moved to External Storage and ES is not accessible to the outside. It's mostly OCR text of PDF files. You can still access them using API (action=query&prop=imageinfo). Nothing to the outside users will change, the API will return the same result, the user interface will show the same thing but it would make all of Wikimedia Commons more reliable and faster to access (by indirect changes such as improving InnoDB buffer pool efficiency), improves time to take database backups, enables us to make bigger changes on image table and improve its schema and much more. I hope no one heavily relies on the img_metadata field in the cloud replicas but if you do, please let me know and reach out for help. You can keep track of the work in https://phabricator.wikimedia.org/T275268 Thank you for understanding and sorry for any inconvenience. -- Amir (he/him)

1 0

Hibernating a VM?
by Roy Smith 07 Jun '21

07 Jun '21

I've got a VM (spi-tools-host-1) that I'm not actively using right now. I could shut it down to release the CPU resources, but I want to be able to start it back up again at some point in the future with the same configuration. On AWS I would just back up all the storage and shut down the VM. Is there some way to do that in this environment? I suspect the answer may include, "Yo, dummy, you should have used puppet instead of just installing things manually with apt-get". If that's the case, feel free to say that :-) Or, is the VM so small that it's not worth worrying too much about the resources it's wasting?

3 5

[Cloud-announce] Mounts for Scratch NFS changing a bit tomorrow (2021-05-27 around 20:00 UTC)
by Brooke Storm 26 May '21

26 May '21

Software that uses /data/scratch may see some disruption tomorrow when I merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/695447/ <https://gerrit.wikimedia.org/r/c/operations/puppet/+/695447/> around 20:00 UTC In order to make the NFS it works on more resilient and healthy, the cluster it runs on is migrating to DRBD failover like Tools home and project uses. The current setup is quite broken. If you are running something against scratch when the patch becomes active in your project, you may see some issues like a stale NFS handle. If that doesn’t resolve quickly, please let us know in Libera.chat: #wikimedia-cloud Brooke Storm Staff SRE Wikimedia Cloud Services bstorm(a)wikimedia.org _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 0

[Cloud-announce] #wikimedia-cloud IRC channel available on libera.chat
by Bryan Davis 26 May '21

26 May '21

TL;DR: * The #wikimedia-cloud IRC channel is moving from Freenode to Libera.Chat. * Register an account on Libera.Chat and join us there! There has been a lot of activity over the last 2-3 days related to staffing changes on the Freenode IRC network [0]. The Wikimedia IRC Group Contacts (GCs) [1] evaluated the situation and decided that moving the Wikimedia IRC channels from Freenode to the brand new Libera.Chat IRC network [2] would be the best course of action [3]. So, we are moving! A new #wikimedia-cloud channel has been created on irc.libera.chat for this Wikimedia sub-community to use. The old channel on Freenode still exists and will be maintained at least until we can get all the bots moved, our documentation updated on wikitech, and we see more folks on the Libera channel than the Freenode one. Messages to our channel in either IRC network, as well as the Telegram channel [4], will be seen on all other channels. There is a new subpage on meta [5] for information on how to create a new account for yourself on Libera.Chat and other related information. There is also a tracking task [6] that you can look at to see various activities that the community hopes to take action on to complete the migration. One last thing: The #wmhack Freenode channel is bridged to #wikimedia-hackathon on Libera.Chat. The new channel name will make it easier for the GCs to help manage spam and other issues that come up occasionally on IRC. Don't miss the fun of our 2021 virtual hackathon from Friday, May 21st to Sunday, May 23rd! [7] [0]: https://www.kline.sh/ [1]: https://meta.wikimedia.org/wiki/IRC/Group_Contacts [2]: https://libera.chat/ [3]: https://meta.wikimedia.org/w/index.php?diff=21476411 [4]: https://t.me/wmcloudirc [5]: https://meta.wikimedia.org/wiki/IRC/Migrating_to_Libera_Chat [6]: https://phabricator.wikimedia.org/T283247 [7]: https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2021 Bryan, on behalf of the WMCS team and the Cloud VPS and Toolforge admins -- Bryan Davis Technical Engagement Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808 _______________________________________________ Wikimedia Cloud Services announce mailing list To unsubscribe send an email to cloud-announce-leave(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)

2 2

2024

2023

2022

2021

2020

2019

2018

2017

Cloud