Starting last Tuesday, I've been getting a persistent error running the
runUpdate.sh script from the Wikidata stand-alone facility.
Example:
06:34:20.402 [update 6] WARN org.wikidata.query.rdf.tool.Update -
Contained error syncing. Giving up on Q13873706
org.wikidata.query.rdf.tool.exception.ContainedException: Unexpected
status code fetching RDF for
https://www.wikidata.org/wiki/Special:EntityData/Q13873706.ttl?nocache=1497…:
429
at
org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.fetchRdfForEntity(WikibaseRepository.java:267)
~[wikidata-query-tools-0.2.3-SNAPSHOT-jar-with-dependencies.jar:na]
at org.wikidata.query.rdf.tool.Update.handleChange(Update.java:474)
~[wikidata-query-tools-0.2.3-SNAPSHOT-jar-with-dependencies.jar:na]
at org.wikidata.query.rdf.tool.Update.access$0(Update.java:472)
~[wikidata-query-tools-0.2.3-SNAPSHOT-jar-with-dependencies.jar:na]
at org.wikidata.query.rdf.tool.Update$1.run(Update.java:370)
~[wikidata-query-tools-0.2.3-SNAPSHOT-jar-with-dependencies.jar:na]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
Does anyone have an explanation for this?
Thanks,
Greetings -
We have three installations of Blazegraph/Wikibase sitting behind a load
balancer. We synchronize each of these servers to the main Wikidata
repository through the wikidata stand-alone facility
(https://www.mediawiki.org/wiki/Wikidata_query_service/User_Manual#Standalon…).
We would like to keep all three servers in sync and current without
duplicating three separate sequences of update requests to the main
Wikidata API. Could anyone provide us with guidance as to how best to do
this? For example is there a straightforward way to set up a complete
installation of the WD API on one of our servers, synchronize that one
to the main WD service, then synchronize our other two Wikibase servers
to our local installation? Alternatively, has anyone done this using
some kind of proxy scheme whose details they could share?
Any help would be appreciated.
Thanks,
Eric Scott
Hi!
I would like to remind everybody that if anything in the code changes
that influences RDF export, this probably influences Wikidata Query
Service (and other RDF consumers) and I would like to ask to do the
following steps when it happens:
1. Send a notification to the wikidata mailing list describing the change
2. Create a phabricator task to update WDQS store
Please note this also concerns adding stuff, not only changing stuff,
since WDQS store may need to be updated when things are added too.
If you are in doubt whether a particular change influences RDF format,
usually if you need to change any test data in
repo/tests/phpunit/data/rdf or repo/tests/phpunit/data/maintenance, then
the change influences RDF. If you don't need to change any tests but you
are sure there's a change in RDF, that means we're missing a test, this
should be brought to Wikidata team's attention too :)
--
Stas Malyshev
smalyshev(a)wikimedia.org
Hello,
This is an important information regarding our database, for people running
tools and scripts.
*The change*
In the wb_terms table, a new column term_full_entity_id containing strings
with the full ID including the letter (ie. “Q42”) will be created, and will
be used instead of the current column term_entity_id that only stores the
numeric part of the ID (ie. “42”).
This change is made, among other things, to support the new entity types to
come, and then to store terms of entities that have non-numeric IDs, for
example Forms (“L42-F1”).
*Implications*
- This change only affects tools that directly access the database.
Other tools, for example those getting data through the API, pywikibot,
gadgets, etc. will not be affected and will work as before with no action
required from the maintainers.
- in order to adapt tools to the new database structure, database
queries using wb_terms should be changed so that they use
term_full_entity_id column instead of term_entity_id. This might
potentially simplify queries. For example, instead of having a
condition WHERE
term_entity_type='item' AND term_entity_id=42, one could now do: WHERE
term_full_entity_id='Q42'.
- term_entity_type is not affected by this change, and will still be
available as before.
*The process*
Starting now, we will work through several steps to achieve this change.
Note that the dates indicated below may not be exact. We will announce each
of the steps separately. Here is a rough timeline:
- June 22th: the new column, term_full_entity_id, becomes visible on
Labs. It will be fully populated in the testwikidatawiki database replica
on Labs. Note that this column will remain incomplete in the wikidatawiki
database until later! Tools that use the wb_terms table can test new
code that uses the new term_full_entity_id column with the
testwikidatawiki database, but must keep using the old code with the
wikidatawiki database.
- Some time later: testwikidatawiki should be fully populated, and
become usable on the wikidatawiki database. Tools that use the wb_terms
table can now switch to using new code that no longer uses the old
term_entity_id column on all databases.
- Eventually (not before July 6th): the old term_entity_id column is
removed from the testwikidatawiki and wikidatawiki database replicas on
labs. Any code that still uses the old term_entity_id will break.
If you have any question or problem regarding this change, feel free to
answer to this mail or write directly to me.
--
Léa Lacroix
Project Manager Community Communication for Wikidata
Wikimedia Deutschland e.V.
Tempelhofer Ufer 23-24
10963 Berlin
www.wikimedia.de
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/029/42207.
Hi everyone,
In September 2017 (15.09-17.09), one of the largest hackathons in Europe
will take place in Zurich, Switzerland. The event is called HackZurich
<http://digitalfestival.ch/en/HACK/> and it is organized together with
the Digital
Festival 2017 <http://digitalfestival.ch/en>.
It is a great opportunity to make open data enthusiasts and hackers in
general aware of Wikidata, and we would also like to team up with other
Wikidata developers to *participate at HackZurich*.
One day before HackZurich, on Thu 14.09 we will be *organising a Wikidata
workshop* at the University of Zurich, where we would like to offer
information and training about Wikidata.
All the information about both the Wikidata workshop and the participation
at HackZurich can be found at the event's page:
https://www.wikidata.org/wiki/Wikidata:Events/Wikidata_Zurich
*Important*: the registration for HackZurich opens on Friday 09.06. The
workshop also requires registration
<https://docs.google.com/forms/d/e/1FAIpQLSd2srUW6LjdnkpmDUKz-SkE20kGxGImX8Z…>
.
If you would like to help out in the workshop or/and participate in the
hackathon, please register in each of the events and let us know in the
Talk page:
https://www.wikidata.org/wiki/Wikidata_talk:Events/Wikidata_Zurich
Thanks!
Cristina Sarasua and Leon Kastler
Hi everyone,
This is an important message to all the people running external tools.
About 2 years ago we said we want to get rid of the
"wb_entity_per_page" table. This is still the case and is becoming
more pressing now with the introduction of support for lexicographical
data.
On July 12th, we are going to stop updating the "wb_entity_per_page"
table from the Wikibase database and stop its replication on ToolLabs.
At a later point we will remove it completely.
"wb_entity_per_page" was a secondary database table, mapping Wikibase
entity IDs (e.g. "Q42") to MediaWiki page IDs (e.g. 138, which can be
seen at https://www.wikidata.org/wiki/Q42?action=info).
"wb_entity_per_page" stored entity IDs as numbers, while page titles
are always full entity IDs.
This mapping existed because Wikibase was designed with the
possibility to have entity pages where the ID does not match the
title. This idea was never used, and finally removed in 2015
(documented in https://phabricator.wikimedia.org/T95685). We decided
to get rid of the table because it contains outdated information that
could mislead users, it costs resources and could conflict in the
future with our new entity types for lexicographical data.
Please check if you are maintaining any code that accesses the
"wb_entity_per_page" table, and replace it with lookups to MediaWiki's
"page" and "redirect" tables.
We will drop the replica of the table on ToolLabs for
test.wikidata.org on June 28th. We will do the same for wikidata.org
on July 12th.
If you have any question or issue, feel free to add a comment on the
ticket (http://phabricator.wikimedia.org/T95685) or to ping me. Thanks
for your understanding
Cheers
Lydia
--
Lydia Pintscher - http://about.me/lydia.pintscher
Product Manager for Wikidata
Wikimedia Deutschland e.V.
Tempelhofer Ufer 23-24
10963 Berlin
www.wikimedia.de
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg
unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das
Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.