Cloud November 2020

cloud@lists.wikimedia.org

22 participants
8 discussions

[Cloud-announce] Wiki Replicas 2020 Redesign
by Joaquin Oltra Hernandez 15 Dec '20

15 Dec '20

TLDR: Wiki Replicas' architecture is being redesigned for stability and performance. Cross database JOINs will not be available and a host connection will only allow querying its associated DB. See [1] <https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign> for more details. Hi! In the interest of making and keeping Wiki Replicas a stable and performant service, a new backend architecture is needed. There is some impact in the features and usage patterns. What should I do? To avoid breaking changes, you can start making the following changes *now*: - Update existing tools to ensure queries are executed against the proper database connection - Eg: If you want to query the `eswiki_p` DB, you must connect to the `eswiki.analytics.db.svc.eqiad.wmflabs` host and `eswiki_p` DB, and not to enwiki or other hosts - Check your existing tools and services queries for cross database JOINs, rewrite the joins in application code - Eg: If you are doing a join across databases, for example joining `enwiki_p` and `eswiki_p`, you will need to query them separately, and filter the results of the separate queries in the code Timeline: - November - December: Early adopter testing - January 2021: Existing and new systems online, transition period starts - February 2021: Old hardware is decommissioned We need your help - If you would like to beta test the new architecture, please let us know and we will reach out to you soon - Sharing examples / descriptions of how a tool or service was updated, writing a common solution or some example code others can utilize and reference, helping others on IRC and the mailing lists If you have questions or need help adapting your code or queries, please contact us [2] <https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication>, or write on the talk page [3] <https://wikitech.wikimedia.org/wiki/Talk:News/Wiki_Replicas_2020_Redesign>. We will be sending reminders, and more specific examples of the changes via email and on the wiki page. For more information see [1] <https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign>. [1]: https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign [2]: https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication [3]: https://wikitech.wikimedia.org/wiki/Talk:News/Wiki_Replicas_2020_Redesign -- Joaquin Oltra Hernandez Developer Advocate - Wikimedia Foundation _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

16 42

PAWS kubernetes upgrade
by Arturo Borrero Gonzalez 30 Nov '20

30 Nov '20

Hi there, we are about to upgrade the kubernetes version that runs PAWS, from 1.6 to 1.17. We don't expect any interruptions major on the service, perhaps only some hiccups when pods are restarted/rescheduled. More information is available in this phabricator ticket: https://phabricator.wikimedia.org/T268669 The operation may take something between 30 minutes and 1 hours, and we are starting soon after I finish sending this email. Please, ping us if you see anything wrong. regards. -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 0

Request for deletion of Wikitech account and change in shell account name
by Muhammad Usman 29 Nov '20

29 Nov '20

Hi All, I had created a Wikitech account(Usmanmuhd) quite some time ago. I forgot the password and when I tried to use the forgot password I wasn't able to so I went ahead and created another account/s. Could someone please help me out with the below request. 1. Delete the other accounts except *Usmanmuhd* which are associated with this email address(me@usmanmuhd)? 2. Change the shell account name for Usmanmuhd since it contains sensitive information as a part of the name. Thanks in advance, Usman

1 0

Save the date! Coolest Tool Award 2020: December 11th, 17:00 UTC
by Joaquin Oltra Hernandez 25 Nov '20

25 Nov '20

Hi Everyone! The second edition of the Coolest Tool Award <https://meta.wikimedia.org/wiki/Coolest_Tool_Award> will happen online on Friday 11 December 2020 at 17:00 UTC[^1]. The awarded tools will be showcased in a virtual event, with broadcasted video and chat channels for socializing. We will send more details and links soon. Save the date, and join us celebrating the great work volunteer developers do for the Wikimedia communities. We hope to see you there! Joaquin, for the Coolest Tool Academy 2020 [^1]: 17:00 UTC is 9:00 PST, 18:00 CEST, 22:30 IST. More timezones in timeanddate.com <https://www.timeanddate.com/worldclock/fixedtime.html?iso=20201211T17> -- Joaquin Oltra Hernandez Developer Advocate - Wikimedia Foundation

1 0

[Cloud-announce] Cloud VPS users, please claim your projects
by Andrew Bogott 18 Nov '20

18 Nov '20

Every year or so the Cloud Services team tries to identify and clean up unused projects and VMs. We do this via an opt-in process: anyone can mark a project as 'in use,' and that project will be preserved for another year. I've created a wiki page the lists all existing projects, here: https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge If you are a VPS user, please visit that page and mark any projects that you use as {{Used}}. Note that it's not necessary for you to be a project admin to mark something -- if you know that you're currently using a resource and want to keep using it, go ahead and mark it accordingly. If you /are/ a project admin, please take a moment to mark which VMs are or aren't used in your projects. When December arrives, I will shut down and begin the process of reclaiming resources from unused projects. If you think you use a VPS project but aren't sure which, I encourage you to poke around on https://tools.wmflabs.org/openstack-browser/ to see what looks familiar. Worst case, just email cloud(a)lists.wikimedia.org with a description of your use case and we'll sort it out there. Exclusive toolforge users are free to ignore this task. Thank you! -Andrew and WMCS team _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

5 8

[Cloud-announce] 2020-11-10 ToolsDB (User databases in Toolforge) read-only downtime
by Brooke Storm 11 Nov '20

11 Nov '20

The ToolsDB service suffered a breakage in replication on 2020-10-27. WMCS has tried to restore replication of data, but that has been unsuccessful so far including doing a dump to rebuild replication without downtime. At this point, we have a new server waiting to become the replica, but to start the replication process, we need to set the database to read-only for a full dump. This could easily take more than an hour. During that entire time, the database will be read-only. We will begin at 1600 UTC and finish when it is done. The database is quite large, but, with it in read-only mode, I hope the backup will not take terribly long. Please see https://phabricator.wikimedia.org/T266587 <https://phabricator.wikimedia.org/T266587> for additional information. Brooke Storm Staff SRE Wikimedia Cloud Services bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org> IRC: bstorm _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

2 5

General CloudVPS network maintenance on 2020-11-09 (today)
by Arturo Borrero Gonzalez 09 Nov '20

09 Nov '20

Hi!, There will be a general CloudVPS network maintenance on 2020-10-09 @ 12:30 UTC. The operation window will last for 1h. During the operation, all cloud services will be inaccessible or intermittently down. This operation affects all CloudVPS projects, including Toolforge, PAWS and Quarry. Services running in the cloud might fail to contact external entities, and connections to ToolsDB, NFS, wiki-replicas or LDAP will be affected as well. The operation we are doing today is a followup to what we did two weeks ago [0], and involves changing the IP addressing of the network that connects the CloudVPS network to the internet. Sorry for the short notice, we couldn't avoid scheduling this to today. regards. [0] https://phabricator.wikimedia.org/T265288 -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 0

[Cloud-announce] 2020-10-27@1600 UTC Database server maintenance for ToolsDB (user-writeable database service)
by Brooke Storm 02 Nov '20

02 Nov '20

On Tuesday 2020-10-27 at 1600 UTC, ToolsDB, the user database service provided by Toolforge (https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#User_databases <https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#User_databases>) will see some service interruption from restarts and will be set to read-only mode during a scheduled failover. The failover must be done to protect data and minimize downtime while the hypervisor goes down for upgrades. Tools may need to restart in order to reconnect during this time and nothing will be able to write to the database tables until failover is completed. After failover, services should resume as normal, and we will announce when it is done. The process will take some time because we must copy over several databases by hand due to three temporarily unreplicated tables being repaired while things are in read-only mode and the four databases that are always unreplicated being copied (https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#ToolsDB_Backups… <https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#ToolsDB_Backups…>). I expect at least half an hour in read-only mode, but things could take longer or shorter time depending on volume of data to copy and issues encountered during the process. Details are on Phabricator at https://phabricator.wikimedia.org/T263679 <https://phabricator.wikimedia.org/T263679> Brooke Storm Staff SRE Wikimedia Cloud Services bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org> IRC: bstorm _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

2 3

2024

2023

2022

2021

2020

2019

2018

2017

Cloud November 2020