Hello all,
This is an important announcement for all the tool builders and maintainers who access Wikidata’s data by *querying directly Labs database replicas*.
In May-June 2019, the Wikidata development team will drop the wb_terms table from the database in favor of a new optimized schema. Over years, this table has become too big, causing various issues.
This change requires the tools using wb_terms to be updated. Developers and maintainers will need to *adapt their code* to the new schema before the migration starts and switch to the new code when the migration starts.
The migration will start on *May 29th*. On May 15th, a test system will be available for you to test your code.
The table being used by plenty of external tools, we are setting up a process to make sure that the change can be done together with the developers and maintainers, without causing issues and broken tools. Most of the documentation and updates will take place on Phabricator:
- In this Phabricator task https://phabricator.wikimedia.org/T221764, you can find a description of the changes and the process, and you can ask for more details or for help in the comments. This is also where updates will be announced if necessary. - On the Tool Builders Migration board https://phabricator.wikimedia.org/tag/wb_terms_-_tool_builders_migration you will find all the details about the migration, how to update your tool https://phabricator.wikimedia.org/T221765, and you can add your own tasks. - If you need to discuss with the Wikidata developers or get more specific help, we set up two dedicated IRC meetings and a session at the Wikimedia hackathon. More information in this task https://phabricator.wikimedia.org/T221764.
We are aware that this change will ask you to make some important changes in your code, and we are willing to help you as much as our resources allow us to. We hope that you will understand that this change is made to avoid bigger issues in the near future.
Note that this change is not impacting Wikibase instances outside of Wikidata. A dedicated migration plan and announcement will follow.
We strongly encourage you to not wait until last minute to make the changes in your code. If you have any question or issue, we will be happy to help. In order to keep the discussions in one place, please ask questions or raise issues directly in the Phabricator task and board.
Thanks for your understanding, Cheers,
Hello all,
This is an update regarding the dates of test environment and migration of wb_terms table replacement solution https://lists.wikimedia.org/pipermail/wikidata/2019-April/012987.html.
Due to various complications that the developers in the Wikidata team have been working on solving over the last few weeks, we unfortunately will have to push the dates for when a test environment for tools builders will be ready, which was supposed to be ready today, and the following dates for starting migration of wb_terms data into the new schema in production.
The new dates are:
- 29th of May: Test environment for tool builders will be ready - 12th of June: Property Terms migration starts - 19th of June: Read property terms from new schema on Wikidata - 26th of June: Item terms migration begins - 3rd of July: Read item terms from one of the two schemas (as explained in this task https://phabricator.wikimedia.org/T221765)
You can find more information regarding those dates and how to prepare for them in this task https://phabricator.wikimedia.org/T221765, and we have dedicated a board https://phabricator.wikimedia.org/project/view/4014 to receive and help with any questions from tool builders that need to update their tools accordingly.
In order to keep all discussions in one place, we kindly ask you to react or ask your questions on Phabricator https://phabricator.wikimedia.org/T221764.
As a reminder, if you want to discuss with the developers, ask questions and get help in order to update your tools, you can join the IRC Mediawiki meeting, today at 15:00 UTC on #wikimedia-tech.
Thanks,
Léa
On Wed, 24 Apr 2019 at 16:30, Léa Lacroix lea.lacroix@wikimedia.de wrote:
Hello all,
This is an important announcement for all the tool builders and maintainers who access Wikidata’s data by *querying directly Labs database replicas*.
In May-June 2019, the Wikidata development team will drop the wb_terms table from the database in favor of a new optimized schema. Over years, this table has become too big, causing various issues.
This change requires the tools using wb_terms to be updated. Developers and maintainers will need to *adapt their code* to the new schema before the migration starts and switch to the new code when the migration starts.
The migration will start on *May 29th*. On May 15th, a test system will be available for you to test your code.
The table being used by plenty of external tools, we are setting up a process to make sure that the change can be done together with the developers and maintainers, without causing issues and broken tools. Most of the documentation and updates will take place on Phabricator:
- In this Phabricator task https://phabricator.wikimedia.org/T221764,
you can find a description of the changes and the process, and you can ask for more details or for help in the comments. This is also where updates will be announced if necessary.
- On the Tool Builders Migration board
https://phabricator.wikimedia.org/tag/wb_terms_-_tool_builders_migration you will find all the details about the migration, how to update your tool https://phabricator.wikimedia.org/T221765, and you can add your own tasks.
- If you need to discuss with the Wikidata developers or get more
specific help, we set up two dedicated IRC meetings and a session at the Wikimedia hackathon. More information in this task https://phabricator.wikimedia.org/T221764.
We are aware that this change will ask you to make some important changes in your code, and we are willing to help you as much as our resources allow us to. We hope that you will understand that this change is made to avoid bigger issues in the near future.
Note that this change is not impacting Wikibase instances outside of Wikidata. A dedicated migration plan and announcement will follow.
We strongly encourage you to not wait until last minute to make the changes in your code. If you have any question or issue, we will be happy to help. In order to keep the discussions in one place, please ask questions or raise issues directly in the Phabricator task and board.
Thanks for your understanding, Cheers, -- Léa Lacroix Project Manager Community Communication for Wikidata
Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.
Hello all,
This is an update regarding the dates of migration of wb_terms table replacement solution in Wikidata production environment.
When we originally planned the last schedule for migration in production, we must have overlooked the planned offsite for SRE (Software Reliability Engineering) team that overlaps with the first week of it (specifically, beginning to migrate property terms on 12th of June). We unfortunately will have to push the dates for starting migration of wb_terms data into the new schema in production another week into the future from the current dates.
The new dates are:
-
19th of June: Property Terms migration starts -
26th of June: Read property terms from new schema on Wikidata -
3rd of July: Item terms migration begins -
10th of July: Read item terms from one of the two schemas (as explained in this task https://phabricator.wikimedia.org/T221765)
You can find more information regarding those dates and how to prepare for them in this task https://phabricator.wikimedia.org/T221765, and we have dedicated a board https://phabricator.wikimedia.org/project/view/4014/ to receive and help with any questions from tool builders that need to update their tools accordingly.
In order to keep all discussions in one place, we kindly ask you to react or ask your questions on Phabricator.
Thanks for your understanding,
Léa
On Wed, 15 May 2019 at 14:52, Léa Lacroix lea.lacroix@wikimedia.de wrote:
Hello all,
This is an update regarding the dates of test environment and migration of wb_terms table replacement solution https://lists.wikimedia.org/pipermail/wikidata/2019-April/012987.html.
Due to various complications that the developers in the Wikidata team have been working on solving over the last few weeks, we unfortunately will have to push the dates for when a test environment for tools builders will be ready, which was supposed to be ready today, and the following dates for starting migration of wb_terms data into the new schema in production.
The new dates are:
- 29th of May: Test environment for tool builders will be ready
- 12th of June: Property Terms migration starts
- 19th of June: Read property terms from new schema on Wikidata
- 26th of June: Item terms migration begins
- 3rd of July: Read item terms from one of the two schemas (as
explained in this task https://phabricator.wikimedia.org/T221765)
You can find more information regarding those dates and how to prepare for them in this task https://phabricator.wikimedia.org/T221765, and we have dedicated a board https://phabricator.wikimedia.org/project/view/4014 to receive and help with any questions from tool builders that need to update their tools accordingly.
In order to keep all discussions in one place, we kindly ask you to react or ask your questions on Phabricator https://phabricator.wikimedia.org/T221764.
As a reminder, if you want to discuss with the developers, ask questions and get help in order to update your tools, you can join the IRC Mediawiki meeting, today at 15:00 UTC on #wikimedia-tech.
Thanks,
Léa
On Wed, 24 Apr 2019 at 16:30, Léa Lacroix lea.lacroix@wikimedia.de wrote:
Hello all,
This is an important announcement for all the tool builders and maintainers who access Wikidata’s data by *querying directly Labs database replicas*.
In May-June 2019, the Wikidata development team will drop the wb_terms table from the database in favor of a new optimized schema. Over years, this table has become too big, causing various issues.
This change requires the tools using wb_terms to be updated. Developers and maintainers will need to *adapt their code* to the new schema before the migration starts and switch to the new code when the migration starts.
The migration will start on *May 29th*. On May 15th, a test system will be available for you to test your code.
The table being used by plenty of external tools, we are setting up a process to make sure that the change can be done together with the developers and maintainers, without causing issues and broken tools. Most of the documentation and updates will take place on Phabricator:
- In this Phabricator task https://phabricator.wikimedia.org/T221764,
you can find a description of the changes and the process, and you can ask for more details or for help in the comments. This is also where updates will be announced if necessary.
- On the Tool Builders Migration board
https://phabricator.wikimedia.org/tag/wb_terms_-_tool_builders_migration you will find all the details about the migration, how to update your tool https://phabricator.wikimedia.org/T221765, and you can add your own tasks.
- If you need to discuss with the Wikidata developers or get more
specific help, we set up two dedicated IRC meetings and a session at the Wikimedia hackathon. More information in this task https://phabricator.wikimedia.org/T221764.
We are aware that this change will ask you to make some important changes in your code, and we are willing to help you as much as our resources allow us to. We hope that you will understand that this change is made to avoid bigger issues in the near future.
Note that this change is not impacting Wikibase instances outside of Wikidata. A dedicated migration plan and announcement will follow.
We strongly encourage you to not wait until last minute to make the changes in your code. If you have any question or issue, we will be happy to help. In order to keep the discussions in one place, please ask questions or raise issues directly in the Phabricator task and board.
Thanks for your understanding, Cheers, -- Léa Lacroix Project Manager Community Communication for Wikidata
Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.
-- Léa Lacroix Project Manager Community Communication for Wikidata
Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.
Hello all,
This is an update regarding the progress and dates of migration of wb_terms table replacement solution in Wikidata production environment.
We have successfully put Wikidata production in the stage where property terms are stored in and written to both the old store (wb_terms table) and the new replacement store. Retrieving property terms is still being done using the old store.
*The previously announced dates are no longer effective.* No changes to tools are needed yet. Tools can continue to read from the old store (wb_terms table) for the moment. There will be a later announcement regarding the date when tools have to switch to reading property terms from the new store.
The next step will be to go to the stage of retrieving property terms from the new store, while we keep storing them in both stores. That step is blocked by a problem we discovered while testing that switch on beta cluster, and we are working on solving it at the moment ( https://phabricator.wikimedia.org/T226008).
As for item terms, and in the light of the new information about switching master node to a better host (https://phabricator.wikimedia.org/T227062) that can actually host the migration of them till the end, we have as well decided to push out item terms migration on hold until after that failover is done and is stable.
The migration of all item terms will take weeks to finish, but it isn’t clear yet how long exactly. We will run it in several stages and there will be separate announcements regarding those stages to announce and inform about how to deal with it, in case it affects your work.
You can find more information regarding those dates and how to prepare for them in https://phabricator.wikimedia.org/T221765 https://phabricator.wikimedia.org/T221765, and we have dedicated https://phabricator.wikimedia.org/project/view/4014 https://phabricator.wikimedia.org/project/view/4014/ to receive and help with any questions from tool builders that need to update their tools accordingly.
(email & wiki) In order to keep all discussions in one place, we kindly ask you to react or ask your questions https://phabricator.wikimedia.org/T221764 .
Thanks,
On Wed, 15 May 2019 at 15:06, Léa Lacroix lea.lacroix@wikimedia.de wrote:
Hello all,
This is an update regarding the dates of test environment and migration of wb_terms table replacement solution https://lists.wikimedia.org/pipermail/wikidata/2019-April/012987.html.
Due to various complications that the developers in the Wikidata team have been working on solving over the last few weeks, we unfortunately will have to push the dates for when a test environment for tools builders will be ready, which was supposed to be ready today, and the following dates for starting migration of wb_terms data into the new schema in production.
The new dates are:
- 29th of May: Test environment for tool builders will be ready
- 12th of June: Property Terms migration starts
- 19th of June: Read property terms from new schema on Wikidata
- 26th of June: Item terms migration begins
- 3rd of July: Read item terms from one of the two schemas (as
explained in this task https://phabricator.wikimedia.org/T221765)
You can find more information regarding those dates and how to prepare for them in this task https://phabricator.wikimedia.org/T221765, and we have dedicated a board https://phabricator.wikimedia.org/project/view/4014 to receive and help with any questions from tool builders that need to update their tools accordingly.
In order to keep all discussions in one place, we kindly ask you to react or ask your questions on Phabricator https://phabricator.wikimedia.org/T221764.
As a reminder, if you want to discuss with the developers, ask questions and get help in order to update your tools, you can join the IRC Mediawiki meeting, today at 15:00 UTC on #wikimedia-tech.
Thanks,
Léa
On Wed, 24 Apr 2019 at 16:30, Léa Lacroix lea.lacroix@wikimedia.de wrote:
Hello all,
This is an important announcement for all the tool builders and maintainers who access Wikidata’s data by *querying directly Labs database replicas*.
In May-June 2019, the Wikidata development team will drop the wb_terms table from the database in favor of a new optimized schema. Over years, this table has become too big, causing various issues.
This change requires the tools using wb_terms to be updated. Developers and maintainers will need to *adapt their code* to the new schema before the migration starts and switch to the new code when the migration starts.
The migration will start on *May 29th*. On May 15th, a test system will be available for you to test your code.
The table being used by plenty of external tools, we are setting up a process to make sure that the change can be done together with the developers and maintainers, without causing issues and broken tools. Most of the documentation and updates will take place on Phabricator:
- In this Phabricator task https://phabricator.wikimedia.org/T221764,
you can find a description of the changes and the process, and you can ask for more details or for help in the comments. This is also where updates will be announced if necessary.
- On the Tool Builders Migration board
https://phabricator.wikimedia.org/tag/wb_terms_-_tool_builders_migration you will find all the details about the migration, how to update your tool https://phabricator.wikimedia.org/T221765, and you can add your own tasks.
- If you need to discuss with the Wikidata developers or get more
specific help, we set up two dedicated IRC meetings and a session at the Wikimedia hackathon. More information in this task https://phabricator.wikimedia.org/T221764.
We are aware that this change will ask you to make some important changes in your code, and we are willing to help you as much as our resources allow us to. We hope that you will understand that this change is made to avoid bigger issues in the near future.
Note that this change is not impacting Wikibase instances outside of Wikidata. A dedicated migration plan and announcement will follow.
We strongly encourage you to not wait until last minute to make the changes in your code. If you have any question or issue, we will be happy to help. In order to keep the discussions in one place, please ask questions or raise issues directly in the Phabricator task and board.
Thanks for your understanding, Cheers, -- Léa Lacroix Project Manager Community Communication for Wikidata
Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.
-- Léa Lacroix Project Manager Community Communication for Wikidata
Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207. _______________________________________________ Wikidata-tech mailing list Wikidata-tech@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-tech
Thank you!
On Fri, Jul 5, 2019 at 10:40 AM Alaa Sarhan alaa.sarhan@wikimedia.de wrote:
Hello all,
This is an update regarding the progress and dates of migration of wb_terms table replacement solution in Wikidata production environment.
We have successfully put Wikidata production in the stage where property terms are stored in and written to both the old store (wb_terms table) and the new replacement store. Retrieving property terms is still being done using the old store.
*The previously announced dates are no longer effective.* No changes to tools are needed yet. Tools can continue to read from the old store (wb_terms table) for the moment. There will be a later announcement regarding the date when tools have to switch to reading property terms from the new store.
The next step will be to go to the stage of retrieving property terms from the new store, while we keep storing them in both stores. That step is blocked by a problem we discovered while testing that switch on beta cluster, and we are working on solving it at the moment ( https://phabricator.wikimedia.org/T226008).
As for item terms, and in the light of the new information about switching master node to a better host (https://phabricator.wikimedia.org/T227062) that can actually host the migration of them till the end, we have as well decided to push out item terms migration on hold until after that failover is done and is stable.
The migration of all item terms will take weeks to finish, but it isn’t clear yet how long exactly. We will run it in several stages and there will be separate announcements regarding those stages to announce and inform about how to deal with it, in case it affects your work.
You can find more information regarding those dates and how to prepare for them in https://phabricator.wikimedia.org/T221765 https://phabricator.wikimedia.org/T221765, and we have dedicated https://phabricator.wikimedia.org/project/view/4014 https://phabricator.wikimedia.org/project/view/4014/ to receive and help with any questions from tool builders that need to update their tools accordingly.
(email & wiki) In order to keep all discussions in one place, we kindly ask you to react or ask your questions https://phabricator.wikimedia.org/T221764 .
Thanks,
On Wed, 15 May 2019 at 15:06, Léa Lacroix lea.lacroix@wikimedia.de wrote:
Hello all,
This is an update regarding the dates of test environment and migration
of wb_terms
table replacement solution https://lists.wikimedia.org/pipermail/wikidata/2019-April/012987.html.
Due to various complications that the developers in the Wikidata team
have
been working on solving over the last few weeks, we unfortunately will
have
to push the dates for when a test environment for tools builders will be ready, which was supposed to be ready today, and the following dates for starting migration of wb_terms data into the new schema in production.
The new dates are:
- 29th of May: Test environment for tool builders will be ready
- 12th of June: Property Terms migration starts
- 19th of June: Read property terms from new schema on Wikidata
- 26th of June: Item terms migration begins
- 3rd of July: Read item terms from one of the two schemas (as
explained in this task https://phabricator.wikimedia.org/T221765)
You can find more information regarding those dates and how to prepare
for
them in this task https://phabricator.wikimedia.org/T221765, and we have dedicated a board https://phabricator.wikimedia.org/project/view/4014 to receive and
help
with any questions from tool builders that need to update their tools accordingly.
In order to keep all discussions in one place, we kindly ask you to react or ask your questions on Phabricator https://phabricator.wikimedia.org/T221764.
As a reminder, if you want to discuss with the developers, ask questions and get help in order to update your tools, you can join the IRC
Mediawiki
meeting, today at 15:00 UTC on #wikimedia-tech.
Thanks,
Léa
On Wed, 24 Apr 2019 at 16:30, Léa Lacroix lea.lacroix@wikimedia.de wrote:
Hello all,
This is an important announcement for all the tool builders and maintainers who access Wikidata’s data by *querying directly Labs database replicas*.
In May-June 2019, the Wikidata development team will drop the wb_terms table from the database in favor of a new optimized schema. Over years, this table has become too big, causing various issues.
This change requires the tools using wb_terms to be updated. Developers and maintainers will need to *adapt their code* to the new schema before the migration starts and switch to the new code when the migration
starts.
The migration will start on *May 29th*. On May 15th, a test system will be available for you to test your code.
The table being used by plenty of external tools, we are setting up a process to make sure that the change can be done together with the developers and maintainers, without causing issues and broken tools.
Most
of the documentation and updates will take place on Phabricator:
- In this Phabricator task <
https://phabricator.wikimedia.org/T221764%3E,
you can find a description of the changes and the process, and you
can ask
for more details or for help in the comments. This is also where
updates
will be announced if necessary.
- On the Tool Builders Migration board
<
https://phabricator.wikimedia.org/tag/wb_terms_-_tool_builders_migration%3E
you will find all the details about the migration, how to update your tool https://phabricator.wikimedia.org/T221765, and you can add your own tasks.
- If you need to discuss with the Wikidata developers or get more
specific help, we set up two dedicated IRC meetings and a session at
the
Wikimedia hackathon. More information in this task https://phabricator.wikimedia.org/T221764.
We are aware that this change will ask you to make some important
changes
in your code, and we are willing to help you as much as our resources
allow
us to. We hope that you will understand that this change is made to
avoid
bigger issues in the near future.
Note that this change is not impacting Wikibase instances outside of Wikidata. A dedicated migration plan and announcement will follow.
We strongly encourage you to not wait until last minute to make the changes in your code. If you have any question or issue, we will be
happy
to help. In order to keep the discussions in one place, please ask questions or raise issues directly in the Phabricator task and board.
Thanks for your understanding, Cheers, -- Léa Lacroix Project Manager Community Communication for Wikidata
Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das
Finanzamt
für Körperschaften I Berlin, Steuernummer 27/029/42207.
-- Léa Lacroix Project Manager Community Communication for Wikidata
Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207. _______________________________________________ Wikidata-tech mailing list Wikidata-tech@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-tech
--
Alaa Sarhan Full Stack Developer
Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin Phone: +49 (0)30 219 158 26-0https://wikimedia.de
Imagine a world in which every single human being can freely share in the sum of all knowledge. Help us to achieve our vision!https://spenden.wikimedia.de
Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207. _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
wikidata-tech@lists.wikimedia.org