Hi,

These changes are not arbitrary, they are a necessity to keep the replicas functioning. With the growth experienced in the last years (commons, wikidata) the current technology used by Wikireplicas is not feasible. I’ve outlined more technical details in a cloud mailing list response: [1]

We have been keeping track of all the code posted that uses cross-wiki joins, and are planning to analyze more from logs. [2] was made to figure out common use cases and possibilities to enable them.

If you need help evaluating how you will be affected or migrating your code please reach out in the #wikimedia-cloud IRC channel, cloud@lists.wikimedia.org mailing list or make a task in Phabricator #Data-Services [3].

[1] https://lists.wikimedia.org/pipermail/cloud/2021-March/001436.html
[2] https://phabricator.wikimedia.org/T215858
[3] https://phabricator.wikimedia.org/project/board/2874/

On Sun, Mar 14, 2021 at 12:15 AM Daniel Schwen <lists@schwen.de> wrote:
This is the next step after disallowing user databases on replicas. It broke some of my tools but I recently rewrote them to move joining logic into my application. I also replicate small amounts of data (e.g. page titles for a subset of pages) into my user db for joins.
I found it quite off-putting at first. Volunteers are effectively forced to do substantial amounts of extra work after creating tools that "work perfectly fine". 
But I can understand the need to provide more scalable infrastructure with the ever growing projects.

On Sat, Mar 13, 2021, 3:46 PM Yetkin Sakal via Cloud <cloud@lists.wikimedia.org> wrote:
I completely agree with Maarten. It would be a step backward to stop supporting cross-database joins on wiki replicas. This is a breaking change and should not be applied unless a feasible solution to the problem is found.
On Saturday, March 13, 2021, 8:17:39 PM GMT+3, Maarten Dammers <maarten@mdammers.nl> wrote:


Hi Joaquin,

Completely nothing was done with the community input about not being able to do cross wiki joins anymore [1]. In the past the WMF would do something with community input. I guess the new strategy is to just give the appearance of community input to legitimize a decision. Nice way to alienate the volunteers.

Maarten

[1] https://lists.wikimedia.org/pipermail/cloud/2020-November/thread.html#1309

On 12-03-2021 19:16, Joaquin Oltra Hernandez wrote:
TLDR:
- Instead of `*.db.svc.eqiad.wmflabs` use `*.db.svc.wikimedia.cloud` to use the new replicas
- Quarry will migrate March 23 to use the new cluster
- In a ~month (April 15) the old cluster will start retiring. See https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#Timeline for more details

Hi everyone,

I'm happy to announce that the new replicas cluster is available for use after a few weeks open for testing.

To use the new cluster, you will only need to change the hostname when connecting to the databases. Instead of `*.db.svc.eqiad.wmflabs` you can use `*.db.svc.wikimedia.cloud`. See:

- https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#New_host_names
- https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#How_can_I_test_the_new_replicas_before_the_switchover?
- https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database


This brings us to the timeline updates:
https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign#Timeline

- March 2021
  - Announce new cluster domains for all users - March 12
  - Migrate Quarry to use the new cluster - March 23
- April 2021
  - PAWS migration - Estimated first week of April
  - Migrate the old cluster to utilize new replication hosts. Replication may stop. - April 15
  - Redirect old hostnames to the new cluster - April 28

Quarry is ready so it will transition first to use the new replicas. This will happen on March 23, at which point we will deploy the changes and update the docs. We are publishing a note on Tech news but if there are other venues where this information would be useful please help us spread the word.

PAWS is in the process of being migrated. We estimate it could be ready to use the new cluster at the beginning of April, and will publish more details as soon as we know.

On April 15 the old cluster will be moved under the new replication hosts, at which point there may be side effects and replication may stop.
We recommend you update your code earlier to use the new hostnames to verify things are working normally.

Not long after, the old hostnames (`*.db.svc.eqiad.wmflabs`) will be redirected to the new ones (`*.db.svc.wikimedia.cloud`), at which point the old cluster will effectively be inaccessible.


If you have questions or need help please reply to cloud@lists.wikimedia.org, join the #wikimedia-cloud IRC channel, or open a ticket in Phabricator with the Data-Services tag.

Thanks,

_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce


_______________________________________________
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud
_______________________________________________
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud
_______________________________________________
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud
_______________________________________________
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud


--
Joaquin Oltra Hernandez
Developer Advocate - Wikimedia Foundation