TLDR: Wiki Replicas' architecture is being redesigned for stability and
performance. Cross database JOINs will not be available and a host
connection will only allow querying its associated DB. See [1]
<https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign> for
more details.
Hi!
In the interest of making and keeping Wiki Replicas a stable and performant
service, a new backend architecture is needed. There is some impact in the
features and usage patterns.
What should I do? To avoid breaking changes, you can start making the
following changes *now*:
- Update existing tools to ensure queries are executed against the proper
database connection
- Eg: If you want to query the `eswiki_p` DB, you must connect to the
`eswiki.analytics.db.svc.eqiad.wmflabs` host and `eswiki_p` DB, and not to
enwiki or other hosts
- Check your existing tools and services queries for cross database JOINs,
rewrite the joins in application code
- Eg: If you are doing a join across databases, for example joining
`enwiki_p` and `eswiki_p`, you will need to query them separately, and
filter the results of the separate queries in the code
Timeline:
- November - December: Early adopter testing
- January 2021: Existing and new systems online, transition period starts
- February 2021: Old hardware is decommissioned
We need your help
- If you would like to beta test the new architecture, please let us know
and we will reach out to you soon
- Sharing examples / descriptions of how a tool or service was updated,
writing a common solution or some example code others can utilize and
reference, helping others on IRC and the mailing lists
If you have questions or need help adapting your code or queries, please
contact us [2]
<https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication>, or
write on the talk page [3]
<https://wikitech.wikimedia.org/wiki/Talk:News/Wiki_Replicas_2020_Redesign>.
We will be sending reminders, and more specific examples of the changes via
email and on the wiki page. For more information see [1]
<https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign>.
[1]: https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign
[2]: https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication
[3]:
https://wikitech.wikimedia.org/wiki/Talk:News/Wiki_Replicas_2020_Redesign
--
Joaquin Oltra Hernandez
Developer Advocate - Wikimedia Foundation
Hi!,
There will be a general CloudVPS network maintenance on 2020-10-09 @ 12:30 UTC.
The operation window will last for 1h. During the operation, all cloud services
will be inaccessible or intermittently down.
This operation affects all CloudVPS projects, including Toolforge, PAWS and
Quarry. Services running in the cloud might fail to contact external entities,
and connections to ToolsDB, NFS, wiki-replicas or LDAP will be affected as well.
The operation we are doing today is a followup to what we did two weeks ago [0],
and involves changing the IP addressing of the network that connects the
CloudVPS network to the internet.
Sorry for the short notice, we couldn't avoid scheduling this to today.
regards.
[0] https://phabricator.wikimedia.org/T265288
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
Hi!
There will be a general CloudVPS network maintenance on 20202-10-29, from 16:00
UTC to 17:00 UTC.
During the operation window, all cloud services might be intermittently down,
inaccessible.
This operation affects all CloudVPS projects, including Toolforge, PAWS and
Quarry. Services running in the cloud might fail to contact external entities,
and connections to ToolsDB, NFS, wiki-replicas or LDAP might be affected as well.
In the best case scenario, the changes (and downtime) will be barely noticed.
The maintenance consist on introducing new hardware equipment in to the CloudVPS
edge network. You can find additional details in Phabricator [0].
regards.
[0] https://phabricator.wikimedia.org/T265288
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
On Tuesday (2020-10-20) we'll be performing upgrades on the hardware
that hosts the postgres database used by the cloud-vps 'maps' project.
This will probably result in that project being unable to serve tiles
for the duration of the upgrade.
Things will be down for an hour or two, starting around 17:00 UTC.
Earlier today a vulnerability was uncovered that might have permitted
unauthorized users to manipulate project membership or create or delete
VMs within cloud-vps. That issue has since been resolved, and there is
currently no evidence that anyone exploited it.
Nevertheless, out of an abundance of caution: if you are a project
admin, please review the members and projectadmins of your project. If
you see any unknown or untrusted users or unexpected instances, remove
them and notify me directly about what you found and what you've done.
Thank you!
-Andrew + the WMCS team
Hi there,
we need to perform some unscheduled keystone maintenance right now.
Authentication to some cloud services, in particular Horizon, might be
interrupted during this maintenance period. We expect such maintenance to don't
last more than 1h.
regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation