Hi everyone, here is an update before Christmas season:
We are diligently working on T260389 Redesign and rebuild the wikireplicas service using a multi-instance architecture https://phabricator.wikimedia.org/T260389. We are running a bit delayed in terms of the timeline, so early adopter testing will happen in January, and we expect to have the new cluster ready in January.
We have collected and continue discussing all the feedback and are trying to get a clear idea of the use cases that won’t be supported by the new architecture. We have T215858 https://phabricator.wikimedia.org/T215858 as a follow up task to the new architecture, and some tasks (T267992 https://phabricator.wikimedia.org/T267992, T268240 https://phabricator.wikimedia.org/T268240, T268242 https://phabricator.wikimedia.org/T268242, T268244 https://phabricator.wikimedia.org/T268244) for impacted tools and use cases.
If you know about use cases or tools that require cross-joins please comment on the task, make a subtask, or reach out to me.
We published an update in Tech News 2020/49 https://meta.wikimedia.org/wiki/Tech/News/2020/49 at the end of November, and are looking for feedback in T268498 Feedback from Quarry and PAWS users, and other wiki editors affected by the new architecture https://phabricator.wikimedia.org/T268498. If you know of Quarry or PAWS users that use the replicas with cross joins, please direct them to the task to post about how they use them.
Thanks to everyone who discussed, created tasks with their use cases, and helped try out migration paths.
On Tue, Nov 10, 2020 at 9:26 PM Joaquin Oltra Hernandez < jhernandez@wikimedia.org> wrote:
TLDR: Wiki Replicas' architecture is being redesigned for stability and performance. Cross database JOINs will not be available and a host connection will only allow querying its associated DB. See [1] https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign for more details.
Hi!
In the interest of making and keeping Wiki Replicas a stable and performant service, a new backend architecture is needed. There is some impact in the features and usage patterns.
What should I do? To avoid breaking changes, you can start making the following changes *now*:
- Update existing tools to ensure queries are executed against the proper
database connection
- Eg: If you want to query the `eswiki_p` DB, you must connect to the
`eswiki.analytics.db.svc.eqiad.wmflabs` host and `eswiki_p` DB, and not to enwiki or other hosts
- Check your existing tools and services queries for cross database JOINs,
rewrite the joins in application code
- Eg: If you are doing a join across databases, for example joining
`enwiki_p` and `eswiki_p`, you will need to query them separately, and filter the results of the separate queries in the code
Timeline:
- November - December: Early adopter testing
- January 2021: Existing and new systems online, transition period starts
- February 2021: Old hardware is decommissioned
We need your help
- If you would like to beta test the new architecture, please let us know
and we will reach out to you soon
- Sharing examples / descriptions of how a tool or service was updated,
writing a common solution or some example code others can utilize and reference, helping others on IRC and the mailing lists
If you have questions or need help adapting your code or queries, please contact us [2] https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication, or write on the talk page [3] https://wikitech.wikimedia.org/wiki/Talk:News/Wiki_Replicas_2020_Redesign .
We will be sending reminders, and more specific examples of the changes via email and on the wiki page. For more information see [1] https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign.
-- Joaquin Oltra Hernandez Developer Advocate - Wikimedia Foundation