- Cloud-announce - lists.wikimedia.org

Upcoming cloud-vps project deletions
by Andrew Bogott 27 Jan '21

27 Jan '21

Hello! The 2020 project opt-in process wrapped up at the end of the year, and we've identified the following projects as abandoned: - asyncwiki - blog - commons-corruption-checker - fastcci - finding-glams - ign2commons - lizenzhinweisgenerator - lta-tracker - meza - ogvjs-integration - puppet - snuggle - wikidata-federation - wikidata-primary-sources-tool - wikidata-realtime-dumps - wikimania-scholarships At the end of this month (2020-01-31) those projects will be deleted along with all related data and VMs. If you know of anyone associated with those projects who is not on this list, please bring this to their attention. And, if you think any of this is in error, please notify me immediately. Thank you! -Andrew + the WMCS team

1 0

Wiki Replicas Redesign update: New cluster ready for testing on February 1st
by Joaquin Oltra Hernandez 21 Jan '21

21 Jan '21

Hi everyone. We have some updates related to T260389 Redesign and rebuild the wikireplicas service using a multi-instance architecture <https://phabricator.wikimedia.org/T260389>. The new cluster will be ready for testing on February 1st, and we need your help. Please subscribe and comment on T272523 Early testing of the multi-instance architecture <https://phabricator.wikimedia.org/T272523> or reach out if you would like to help test it by migrating your code. We want to closely monitor it and tune it as we ramp up usage. PAWS and Quarry will migrate to use the new cluster later in February, we will publish more information as soon as possible. The old cluster will remain available in parallel during the migration. Thank you for your help, -- Joaquin Oltra Hernandez Developer Advocate - Wikimedia Foundation

1 0

cloud-vps maintenance scheduled for Thursday, 15:00 UTC
by Andrew Bogott 15 Jan '21

15 Jan '21

This Thursday we will be upgrading the cloud-vps OpenStack install to version 'Stein'. During the upgrade window (probably about an hour), Horizon will be disabled. Existing tools and VMs should be largely unaffected, although there may be a brief network interruption when the routing services restart. The upgrade is scheduled for 15:00 UTC, which is 7AM in California. -Andrew + the WMCS team

1 2

Toolforge and Cloud VPS NFS maintenance today
by Brooke Storm 23 Dec '20

23 Dec '20

We will be failing over the Toolforge and Project NFS in 10 minutes to move the main interface to 10Gb Ethernet. The previous work should make this fairly non-disruptive, but that was believed in the past as well. Brooke Storm Cloud Service Team

1 2

ToolsDB Maintenance - 2020-12-16 @ 1700 UTC
by Brooke Storm 17 Dec '20

17 Dec '20

In yet another effort to restore replication and preserve the redundancy of the data in ToolsDB (user writable database in Toolforge), we need to take the database (tools.db.svc.eqiad.wmflabs) completely offline at 1700 UTC on 16 Dec. Apps that depend on the ToolsDB service will fail during the outage (which will take at least an hour, and we aren’t entirely sure exactly how long—expect multiple hours). This will be much faster than the last outage because we are doing a straight copy of the binary database files between the servers. Details of this mess and efforts to restore the replication service can be found at https://phabricator.wikimedia.org/T266587 <https://phabricator.wikimedia.org/T266587> If we succeed in producing a viable copy of the database on another system, we will also perform an upgrade on the hypervisor it is on before closing the maintenance period. That should be an additional hour or so. We appreciate your patience with this process. It is very important that we establish a second copy of this database, especially in light of recent crashes (https://phabricator.wikimedia.org/T253738 <https://phabricator.wikimedia.org/T253738>). Brooke Storm Staff SRE Wikimedia Cloud Services bstorm(a)wikimedia.org IRC: bstorm_

1 2

Toolforge kubernetes maintenance today 2020-12-10 @ 15:30 UTC
by Arturo Borrero Gonzalez 11 Dec '20

11 Dec '20

Hi there! Today 2020-12-10 @ 15:30 UTC we will perform an upgrade of the Toolforge kubernetes cluster [0]. We don't expect any major disruption of the service, but we detected in past upgrades that some components might be restarted, causing brief interruptions of network flows. Given the amount of worker nodes we have, more than 50, the operation will take us at least a couple of hours. Tools maintainers: you don't have to do anything during this operation, but if you detect anything weird please contact us either in the phabricator task [0], in the IRC channel #wikimedia-cloud or in the cloud(a)lists.wikimedia.org [1] mailing list. regards. [0] https://phabricator.wikimedia.org/T263284 [1] https://lists.wikimedia.org/mailman/listinfo/cloud -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

3 3

PAWS kubernetes upgrade
by Arturo Borrero Gonzalez 30 Nov '20

30 Nov '20

Hi there, we are about to upgrade the kubernetes version that runs PAWS, from 1.6 to 1.17. We don't expect any interruptions major on the service, perhaps only some hiccups when pods are restarted/rescheduled. More information is available in this phabricator ticket: https://phabricator.wikimedia.org/T268669 The operation may take something between 30 minutes and 1 hours, and we are starting soon after I finish sending this email. Please, ping us if you see anything wrong. regards. -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 0

Cloud VPS users, please claim your projects
by Andrew Bogott 19 Nov '20

19 Nov '20

Every year or so the Cloud Services team tries to identify and clean up unused projects and VMs. We do this via an opt-in process: anyone can mark a project as 'in use,' and that project will be preserved for another year. I've created a wiki page the lists all existing projects, here: https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge If you are a VPS user, please visit that page and mark any projects that you use as {{Used}}. Note that it's not necessary for you to be a project admin to mark something -- if you know that you're currently using a resource and want to keep using it, go ahead and mark it accordingly. If you /are/ a project admin, please take a moment to mark which VMs are or aren't used in your projects. When December arrives, I will shut down and begin the process of reclaiming resources from unused projects. If you think you use a VPS project but aren't sure which, I encourage you to poke around on https://tools.wmflabs.org/openstack-browser/ to see what looks familiar. Worst case, just email cloud(a)lists.wikimedia.org with a description of your use case and we'll sort it out there. Exclusive toolforge users are free to ignore this task. Thank you! -Andrew and WMCS team

1 3

2020-11-10 ToolsDB (User databases in Toolforge) read-only downtime
by Brooke Storm 12 Nov '20

12 Nov '20

The ToolsDB service suffered a breakage in replication on 2020-10-27. WMCS has tried to restore replication of data, but that has been unsuccessful so far including doing a dump to rebuild replication without downtime. At this point, we have a new server waiting to become the replica, but to start the replication process, we need to set the database to read-only for a full dump. This could easily take more than an hour. During that entire time, the database will be read-only. We will begin at 1600 UTC and finish when it is done. The database is quite large, but, with it in read-only mode, I hope the backup will not take terribly long. Please see https://phabricator.wikimedia.org/T266587 <https://phabricator.wikimedia.org/T266587> for additional information. Brooke Storm Staff SRE Wikimedia Cloud Services bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org> IRC: bstorm

1 4

Wiki Replicas 2020 Redesign
by Joaquin Oltra Hernandez 11 Nov '20

11 Nov '20

TLDR: Wiki Replicas' architecture is being redesigned for stability and performance. Cross database JOINs will not be available and a host connection will only allow querying its associated DB. See [1] <https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign> for more details. Hi! In the interest of making and keeping Wiki Replicas a stable and performant service, a new backend architecture is needed. There is some impact in the features and usage patterns. What should I do? To avoid breaking changes, you can start making the following changes *now*: - Update existing tools to ensure queries are executed against the proper database connection - Eg: If you want to query the `eswiki_p` DB, you must connect to the `eswiki.analytics.db.svc.eqiad.wmflabs` host and `eswiki_p` DB, and not to enwiki or other hosts - Check your existing tools and services queries for cross database JOINs, rewrite the joins in application code - Eg: If you are doing a join across databases, for example joining `enwiki_p` and `eswiki_p`, you will need to query them separately, and filter the results of the separate queries in the code Timeline: - November - December: Early adopter testing - January 2021: Existing and new systems online, transition period starts - February 2021: Old hardware is decommissioned We need your help - If you would like to beta test the new architecture, please let us know and we will reach out to you soon - Sharing examples / descriptions of how a tool or service was updated, writing a common solution or some example code others can utilize and reference, helping others on IRC and the mailing lists If you have questions or need help adapting your code or queries, please contact us [2] <https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication>, or write on the talk page [3] <https://wikitech.wikimedia.org/wiki/Talk:News/Wiki_Replicas_2020_Redesign>. We will be sending reminders, and more specific examples of the changes via email and on the wiki page. For more information see [1] <https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign>. [1]: https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign [2]: https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication [3]: https://wikitech.wikimedia.org/wiki/Talk:News/Wiki_Replicas_2020_Redesign -- Joaquin Oltra Hernandez Developer Advocate - Wikimedia Foundation

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-announce