Wikidata-tech June 2017

wikidata-tech@lists.wikimedia.org

9 participants
8 discussions

(no subject)
by zitouni noureddine 16 Jun '17

16 Jun '17

1 0

org.wikidata.query.rdf.tool.Update-Contained error syncing. Giving up on Qxxxxx
by Eric Scott 15 Jun '17

15 Jun '17

Starting last Tuesday, I've been getting a persistent error running the runUpdate.sh script from the Wikidata stand-alone facility. Example: 06:34:20.402 [update 6] WARN org.wikidata.query.rdf.tool.Update - Contained error syncing. Giving up on Q13873706 org.wikidata.query.rdf.tool.exception.ContainedException: Unexpected status code fetching RDF for https://www.wikidata.org/wiki/Special:EntityData/Q13873706.ttl?nocache=1497…: 429 at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.fetchRdfForEntity(WikibaseRepository.java:267) ~[wikidata-query-tools-0.2.3-SNAPSHOT-jar-with-dependencies.jar:na] at org.wikidata.query.rdf.tool.Update.handleChange(Update.java:474) ~[wikidata-query-tools-0.2.3-SNAPSHOT-jar-with-dependencies.jar:na] at org.wikidata.query.rdf.tool.Update.access$0(Update.java:472) ~[wikidata-query-tools-0.2.3-SNAPSHOT-jar-with-dependencies.jar:na] at org.wikidata.query.rdf.tool.Update$1.run(Update.java:370) ~[wikidata-query-tools-0.2.3-SNAPSHOT-jar-with-dependencies.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_131] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131] Does anyone have an explanation for this? Thanks,

2 1

Synchronizing multiple copies of Wikibase to the WD mothership efficiently
by Eric Scott 13 Jun '17

13 Jun '17

Greetings - We have three installations of Blazegraph/Wikibase sitting behind a load balancer. We synchronize each of these servers to the main Wikidata repository through the wikidata stand-alone facility (https://www.mediawiki.org/wiki/Wikidata_query_service/User_Manual#Standalon…). We would like to keep all three servers in sync and current without duplicating three separate sequences of update requests to the main Wikidata API. Could anyone provide us with guidance as to how best to do this? For example is there a straightforward way to set up a complete installation of the WD API on one of our servers, synchronize that one to the main WD service, then synchronize our other two Wikibase servers to our local installation? Alternatively, has anyone done this using some kind of proxy scheme whose details they could share? Any help would be appreciated. Thanks, Eric Scott

1 0

REMINDER about changing RDF formats
by Stas Malyshev 13 Jun '17

13 Jun '17

Hi! I would like to remind everybody that if anything in the code changes that influences RDF export, this probably influences Wikidata Query Service (and other RDF consumers) and I would like to ask to do the following steps when it happens: 1. Send a notification to the wikidata mailing list describing the change 2. Create a phabricator task to update WDQS store Please note this also concerns adding stuff, not only changing stuff, since WDQS store may need to be updated when things are added too. If you are in doubt whether a particular change influences RDF format, usually if you need to change any test data in repo/tests/phpunit/data/rdf or repo/tests/phpunit/data/maintenance, then the change influences RDF. If you don't need to change any tests but you are sure there's a change in RDF, that means we're missing a test, this should be brought to Wikidata team's attention too :) -- Stas Malyshev smalyshev(a)wikimedia.org

1 0

Breaking change: improving the schema of wb_terms table
by Léa Lacroix 08 Jun '17

08 Jun '17

Hello, This is an important information regarding our database, for people running tools and scripts. *The change* In the wb_terms table, a new column term_full_entity_id containing strings with the full ID including the letter (ie. “Q42”) will be created, and will be used instead of the current column term_entity_id that only stores the numeric part of the ID (ie. “42”). This change is made, among other things, to support the new entity types to come, and then to store terms of entities that have non-numeric IDs, for example Forms (“L42-F1”). *Implications* - This change only affects tools that directly access the database. Other tools, for example those getting data through the API, pywikibot, gadgets, etc. will not be affected and will work as before with no action required from the maintainers. - in order to adapt tools to the new database structure, database queries using wb_terms should be changed so that they use term_full_entity_id column instead of term_entity_id. This might potentially simplify queries. For example, instead of having a condition WHERE term_entity_type='item' AND term_entity_id=42, one could now do: WHERE term_full_entity_id='Q42'. - term_entity_type is not affected by this change, and will still be available as before. *The process* Starting now, we will work through several steps to achieve this change. Note that the dates indicated below may not be exact. We will announce each of the steps separately. Here is a rough timeline: - June 22th: the new column, term_full_entity_id, becomes visible on Labs. It will be fully populated in the testwikidatawiki database replica on Labs. Note that this column will remain incomplete in the wikidatawiki database until later! Tools that use the wb_terms table can test new code that uses the new term_full_entity_id column with the testwikidatawiki database, but must keep using the old code with the wikidatawiki database. - Some time later: testwikidatawiki should be fully populated, and become usable on the wikidatawiki database. Tools that use the wb_terms table can now switch to using new code that no longer uses the old term_entity_id column on all databases. - Eventually (not before July 6th): the old term_entity_id column is removed from the testwikidatawiki and wikidatawiki database replicas on labs. Any code that still uses the old term_entity_id will break. If you have any question or problem regarding this change, feel free to answer to this mail or write directly to me. -- Léa Lacroix Project Manager Community Communication for Wikidata Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

1 0

Wikidata Zurich: Workshop and Participating at HackZurich (14.09, 15.09-17.09)
by wikidata zurich 06 Jun '17

06 Jun '17

Hi everyone, In September 2017 (15.09-17.09), one of the largest hackathons in Europe will take place in Zurich, Switzerland. The event is called HackZurich <http://digitalfestival.ch/en/HACK/> and it is organized together with the Digital Festival 2017 <http://digitalfestival.ch/en>. It is a great opportunity to make open data enthusiasts and hackers in general aware of Wikidata, and we would also like to team up with other Wikidata developers to *participate at HackZurich*. One day before HackZurich, on Thu 14.09 we will be *organising a Wikidata workshop* at the University of Zurich, where we would like to offer information and training about Wikidata. All the information about both the Wikidata workshop and the participation at HackZurich can be found at the event's page: https://www.wikidata.org/wiki/Wikidata:Events/Wikidata_Zurich *Important*: the registration for HackZurich opens on Friday 09.06. The workshop also requires registration <https://docs.google.com/forms/d/e/1FAIpQLSd2srUW6LjdnkpmDUKz-SkE20kGxGImX8Z…> . If you would like to help out in the workshop or/and participate in the hackathon, please register in each of the events and let us know in the Talk page: https://www.wikidata.org/wiki/Wikidata_talk:Events/Wikidata_Zurich Thanks! Cristina Sarasua and Leon Kastler

1 0

RDF to wikibase
by Dennis Diefenbach 05 Jun '17

05 Jun '17

Hello everyone, first, thank you for providing Wikibase!!!! We have put up a local Wikibase service, and we are interesting to import in it RDF. We figured out that one option is using the wikibase APIs but we thought that maybe someone already had this problem. We found the following project https://github.com/rdfio/RDFIO <https://github.com/rdfio/RDFIO>, but if we got it right it does not fit. Any ideas, pointers? Thank you Dennis Diefenbach <http://laboratoirehubertcurien.fr/> Dennis DIEFENBACH Doctorand dennis.diefenbach(a)univ-st-etienne.fr <mailto:dennis.diefenbach@univ-st-etienne.fr> Laboratoire Hubert Curien Campus Manufacture 18 RUE Pr Benoît Lauras 42000 SAINT-ETIENNE 04 77 91 57 80 http://laboratoirehubertcurien.fr <http://laboratoirehubertcurien.fr/> <http://www.univ-st-etienne.fr/> <http://www.facebook.com/Universite.Jean.Monnet.Saint.Etienne> <https://twitter.com/Univ_St_Etienne> <https://www.youtube.com/user/UniJeanMonnetUJM> <https://www.linkedin.com/edu/universit%C3%A9-jean-monnet-saint-etienne-12533> <http://www.viadeo.com/fr/company/universite-jean-monnet> <http://www.universite-lyon.fr/>

1 0

Breaking change: "wb_entity_per_page" table will not be updated and replicated on ToolLabs anymore
by Lydia Pintscher 01 Jun '17

01 Jun '17

Hi everyone, This is an important message to all the people running external tools. About 2 years ago we said we want to get rid of the "wb_entity_per_page" table. This is still the case and is becoming more pressing now with the introduction of support for lexicographical data. On July 12th, we are going to stop updating the "wb_entity_per_page" table from the Wikibase database and stop its replication on ToolLabs. At a later point we will remove it completely. "wb_entity_per_page" was a secondary database table, mapping Wikibase entity IDs (e.g. "Q42") to MediaWiki page IDs (e.g. 138, which can be seen at https://www.wikidata.org/wiki/Q42?action=info). "wb_entity_per_page" stored entity IDs as numbers, while page titles are always full entity IDs. This mapping existed because Wikibase was designed with the possibility to have entity pages where the ID does not match the title. This idea was never used, and finally removed in 2015 (documented in https://phabricator.wikimedia.org/T95685). We decided to get rid of the table because it contains outdated information that could mislead users, it costs resources and could conflict in the future with our new entity types for lexicographical data. Please check if you are maintaining any code that accesses the "wb_entity_per_page" table, and replace it with lookups to MediaWiki's "page" and "redirect" tables. We will drop the replica of the table on ToolLabs for test.wikidata.org on June 28th. We will do the same for wikidata.org on July 12th. If you have any question or issue, feel free to add a comment on the ticket (http://phabricator.wikimedia.org/T95685) or to ping me. Thanks for your understanding Cheers Lydia -- Lydia Pintscher - http://about.me/lydia.pintscher Product Manager for Wikidata Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

3 5

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

Wikidata-tech June 2017