It might be a good idea to add an API that outputs the entity IDs that changed since time x or revision y. For older data it could refer to index files for the dumps. Makes probably more sense than to create a dump each minute.
hourly dumps: https://phabricator.wikimedia.org/T85100 changed entities API: https://phabricator.wikimedia.org/T85103
On Sat, Dec 20, 2014 at 9:03 PM, Stas Malyshev smalyshev@wikimedia.org wrote:
Hi!
The best place for this kind of question would be the wikidata-tech
mailing list
wikidata-tech@lists.wikimedia.org. It would probably be a good idea
if you
(and whoever else deals with wikidata on the technical level) were
subscribed
there. It's pretty low traffic.
Thanks, I've sent the subscription request and adding it to the CC. Still learning the right places to go for things :)
Statement IDs are GUIDs (with the Item ID prefixed), and they do not
change when
the Statement changes (otherwise, they would be hashes, not IDs -
References are
currently handled by hash).
From the export/import point of view, I think I'd prefer immutable claims (i.e. ID changes each time claim changes) as they are easier to handle, but as it is not the case, I can switch to using the content hash instead. The performance impact (time spent calculating the hashes) should not be too big.
One thing that would be rather easy to do is to make JSON dumps of just
the
items that changed in the last X hours. But that wouldn't tell you wich statements changed.
I think for imports the best thing would be to have real diffs - i.e. list of claims/item fields that were added/removed/changed - but if that's not feasible, list of changed items would be great too. We may want this with even more frequency than hours. Item data is not that big, so loading it and running the diff manually would still be workable. It would be slightly slower for big items (since each claim for the item has to be examined) and requires maintaining additional data structure to efficiently enumerate the claims, but it should be still workable.
Thanks, Stas