[Breaking change] Important for Wikidata tools maintainers: wb_terms table to be dropped at the end of May

List overview All Threads
Download

newer

older

Queries for assessing Wikidata

No Wikidata JSON dumps for the...

Léa Lacroix

24 Apr 2019 24 Apr '19

2:30 p.m.

Hello all, This is an important announcement for all the tool builders and maintainers who access Wikidata’s data by *querying directly Labs database replicas*. In May-June 2019, the Wikidata development team will drop the wb_terms table from the database in favor of a new optimized schema. Over years, this table has become too big, causing various issues. This change requires the tools using wb_terms to be updated. Developers and maintainers will need to *adapt their code* to the new schema before the migration starts and switch to the new code when the migration starts. The migration will start on *May 29th*. On May 15th, a test system will be available for you to test your code. The table being used by plenty of external tools, we are setting up a process to make sure that the change can be done together with the developers and maintainers, without causing issues and broken tools. Most of the documentation and updates will take place on Phabricator: - In this Phabricator task <https://phabricator.wikimedia.org/T221764>, you can find a description of the changes and the process, and you can ask for more details or for help in the comments. This is also where updates will be announced if necessary. - On the Tool Builders Migration board <https://phabricator.wikimedia.org/tag/wb_terms_-_tool_builders_migration> you will find all the details about the migration, how to update your tool <https://phabricator.wikimedia.org/T221765>, and you can add your own tasks. - If you need to discuss with the Wikidata developers or get more specific help, we set up two dedicated IRC meetings and a session at the Wikimedia hackathon. More information in this task <https://phabricator.wikimedia.org/T221764>. We are aware that this change will ask you to make some important changes in your code, and we are willing to help you as much as our resources allow us to. We hope that you will understand that this change is made to avoid bigger issues in the near future. Note that this change is not impacting Wikibase instances outside of Wikidata. A dedicated migration plan and announcement will follow. We strongly encourage you to not wait until last minute to make the changes in your code. If you have any question or issue, we will be happy to help. In order to keep the discussions in one place, please ask questions or raise issues directly in the Phabricator task and board. Thanks for your understanding, Cheers, -- Léa Lacroix Project Manager Community Communication for Wikidata Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

Attachments:

attachment.htm (text/html — 3.4 KB)

Show replies by thread

Léa Lacroix

15 May 15 May

12:52 p.m.

Hello all, This is an update regarding the dates of test environment and migration of wb_terms table replacement solution <https://lists.wikimedia.org/pipermail/wikidata/2019-April/012987.html>. Due to various complications that the developers in the Wikidata team have been working on solving over the last few weeks, we unfortunately will have to push the dates for when a test environment for tools builders will be ready, which was supposed to be ready today, and the following dates for starting migration of wb_terms data into the new schema in production. The new dates are: - 29th of May: Test environment for tool builders will be ready - 12th of June: Property Terms migration starts - 19th of June: Read property terms from new schema on Wikidata - 26th of June: Item terms migration begins - 3rd of July: Read item terms from one of the two schemas (as explained in this task <https://phabricator.wikimedia.org/T221765>) You can find more information regarding those dates and how to prepare for them in this task <https://phabricator.wikimedia.org/T221765>, and we have dedicated a board <https://phabricator.wikimedia.org/project/view/4014> to receive and help with any questions from tool builders that need to update their tools accordingly. In order to keep all discussions in one place, we kindly ask you to react or ask your questions on Phabricator <https://phabricator.wikimedia.org/T221764>. As a reminder, if you want to discuss with the developers, ask questions and get help in order to update your tools, you can join the IRC Mediawiki meeting, today at 15:00 UTC on #wikimedia-tech. Thanks, Léa On Wed, 24 Apr 2019 at 16:30, Léa Lacroix <lea.lacroix(a)wikimedia.de> wrote:

...

-- Léa Lacroix Project Manager Community Communication for Wikidata Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

Léa Lacroix

11 Jun 11 Jun

8:06 a.m.

New subject: [Breaking change] Important for Wikidata tools maintainers: wb_terms table to be dropped

Hello all, This is an update regarding the dates of migration of wb_terms table replacement solution in Wikidata production environment. When we originally planned the last schedule for migration in production, we must have overlooked the planned offsite for SRE (Software Reliability Engineering) team that overlaps with the first week of it (specifically, beginning to migrate property terms on 12th of June). We unfortunately will have to push the dates for starting migration of wb_terms data into the new schema in production another week into the future from the current dates. The new dates are: - 19th of June: Property Terms migration starts - 26th of June: Read property terms from new schema on Wikidata - 3rd of July: Item terms migration begins - 10th of July: Read item terms from one of the two schemas (as explained in this task <https://phabricator.wikimedia.org/T221765>) You can find more information regarding those dates and how to prepare for them in this task <https://phabricator.wikimedia.org/T221765>, and we have dedicated a board <https://phabricator.wikimedia.org/project/view/4014/> to receive and help with any questions from tool builders that need to update their tools accordingly. In order to keep all discussions in one place, we kindly ask you to react or ask your questions on Phabricator. Thanks for your understanding, Léa On Wed, 15 May 2019 at 14:52, Léa Lacroix <lea.lacroix(a)wikimedia.de> wrote:

...

Alaa Sarhan

5 Jul 5 Jul

10:39 a.m.

New subject: [Wikidata-tech] [Breaking change] Important for Wikidata tools maintainers: wb_terms table to be dropped at the end of May

Hello all, This is an update regarding the progress and dates of migration of wb_terms table replacement solution in Wikidata production environment. We have successfully put Wikidata production in the stage where property terms are stored in and written to both the old store (wb_terms table) and the new replacement store. Retrieving property terms is still being done using the old store. *The previously announced dates are no longer effective.* No changes to tools are needed yet. Tools can continue to read from the old store (wb_terms table) for the moment. There will be a later announcement regarding the date when tools have to switch to reading property terms from the new store. The next step will be to go to the stage of retrieving property terms from the new store, while we keep storing them in both stores. That step is blocked by a problem we discovered while testing that switch on beta cluster, and we are working on solving it at the moment ( https://phabricator.wikimedia.org/T226008). As for item terms, and in the light of the new information about switching master node to a better host (https://phabricator.wikimedia.org/T227062) that can actually host the migration of them till the end, we have as well decided to push out item terms migration on hold until after that failover is done and is stable. The migration of all item terms will take weeks to finish, but it isn’t clear yet how long exactly. We will run it in several stages and there will be separate announcements regarding those stages to announce and inform about how to deal with it, in case it affects your work. You can find more information regarding those dates and how to prepare for them in https://phabricator.wikimedia.org/T221765 <https://phabricator.wikimedia.org/T221765>, and we have dedicated https://phabricator.wikimedia.org/project/view/4014 <https://phabricator.wikimedia.org/project/view/4014/> to receive and help with any questions from tool builders that need to update their tools accordingly. (email & wiki) In order to keep all discussions in one place, we kindly ask you to react or ask your questions https://phabricator.wikimedia.org/T221764 . Thanks, On Wed, 15 May 2019 at 15:06, Léa Lacroix <lea.lacroix(a)wikimedia.de> wrote:

...

-- Alaa Sarhan Full Stack Developer Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin Phone: +49 (0)30 219 158 26-0https://wikimedia.de Imagine a world in which every single human being can freely share in the sum of all knowledge. Help us to achieve our vision!https://spenden.wikimedia.de Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

Magnus Manske

12:07 p.m.

New subject: [Wikitech-l] [Wikidata-tech] [Breaking change] Important for Wikidata tools maintainers: wb_terms table to be dropped at the end of May

Thank you! On Fri, Jul 5, 2019 at 10:40 AM Alaa Sarhan <alaa.sarhan(a)wikimedia.de> wrote:

...

Hello all, This is an update regarding the dates of test environment and migration

of wb_terms

table replacement solution <https://lists.wikimedia.org/pipermail/wikidata/2019-April/012987.html>. Due to various complications that the developers in the Wikidata team

have

been working on solving over the last few weeks, we unfortunately will

have

to push the dates for when a test environment for tools builders will be ready, which was supposed to be ready today, and the following dates for starting migration of wb_terms data into the new schema in production. The new dates are: - 29th of May: Test environment for tool builders will be ready - 12th of June: Property Terms migration starts - 19th of June: Read property terms from new schema on Wikidata - 26th of June: Item terms migration begins - 3rd of July: Read item terms from one of the two schemas (as explained in this task <https://phabricator.wikimedia.org/T221765>) You can find more information regarding those dates and how to prepare

for

them in this task <https://phabricator.wikimedia.org/T221765>, and we have dedicated a board <https://phabricator.wikimedia.org/project/view/4014> to receive and

help

with any questions from tool builders that need to update their tools accordingly. In order to keep all discussions in one place, we kindly ask you to react or ask your questions on Phabricator <https://phabricator.wikimedia.org/T221764>. As a reminder, if you want to discuss with the developers, ask questions and get help in order to update your tools, you can join the IRC

Mediawiki

meeting, today at 15:00 UTC on #wikimedia-tech. Thanks, Léa On Wed, 24 Apr 2019 at 16:30, Léa Lacroix <lea.lacroix(a)wikimedia.de> wrote: > Hello all, > > This is an important announcement for all the tool builders and > maintainers who access Wikidata’s data by *querying directly Labs > database replicas*. > > In May-June 2019, the Wikidata development team will drop the wb_terms > table from the database in favor of a new optimized schema. Over years, > this table has become too big, causing various issues. > > This change requires the tools using wb_terms to be updated. Developers > and maintainers will need to *adapt their code* to the new schema before > the migration starts and switch to the new code when the migration

starts.

> > The migration will start on *May 29th*. On May 15th, a test system will > be available for you to test your code. > > The table being used by plenty of external tools, we are setting up a > process to make sure that the change can be done together with the > developers and maintainers, without causing issues and broken tools.

Most

> of the documentation and updates will take place on Phabricator: > > - In this Phabricator task <

https://phabricator.wikimedia.org/T221764>gt;,

> you can find a description of the changes and the process, and you

can ask

> for more details or for help in the comments. This is also where

updates

> will be announced if necessary. > - On the Tool Builders Migration board > <

https://phabricator.wikimedia.org/tag/wb_terms_-_tool_builders_migration>

> you will find all the details about the migration, how to update your > tool <https://phabricator.wikimedia.org/T221765>, and you can add > your own tasks. > - If you need to discuss with the Wikidata developers or get more > specific help, we set up two dedicated IRC meetings and a session at

the

> Wikimedia hackathon. More information in this task > <https://phabricator.wikimedia.org/T221764>. > > We are aware that this change will ask you to make some important

changes

> in your code, and we are willing to help you as much as our resources

allow

> us to. We hope that you will understand that this change is made to

avoid

> bigger issues in the near future. > > Note that this change is not impacting Wikibase instances outside of > Wikidata. A dedicated migration plan and announcement will follow. > > We strongly encourage you to not wait until last minute to make the > changes in your code. If you have any question or issue, we will be

happy

> to help. In order to keep the discussions in one place, please ask > questions or raise issues directly in the Phabricator task and board. > > Thanks for your understanding, > Cheers, > -- > Léa Lacroix > Project Manager Community Communication for Wikidata > > Wikimedia Deutschland e.V. > Tempelhofer Ufer 23-24 > 10963 Berlin > www.wikimedia.de > > Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. > > Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg > unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das

Finanzamt

für Körperschaften I Berlin, Steuernummer 27/029/42207.

1757

days inactive

1829

days old

wikidata@lists.wikimedia.org

Manage subscription

4 comments

3 participants

tags (0)

participants (3)

Alaa Sarhan
Léa Lacroix
Magnus Manske