Hi!
On 5/15/18 3:27 PM, Justin Maltais wrote:
Hi,
I am looking for the most efficient way of getting the following
information out of WDQS:
* One language only (e.g.
fr.wikipedia.org)
* All instances of human (e.g. of the abstraction: wd:Q9916|Dwight
David
Let's say we have a list of all sovereign states
(Q16, Q30, Q142, ...)
and all letters of the requested language (French: a, b, c, ...) , we
can automate requests and get a lot of results. Unfortunately, it's
costly and not efficient. It takes about a day to succeed.
The first thing I would like to ask is please don't do that again. This
created a significant load on the server, the script completely ignored
the throttling headers we sent, and in the future we would ban such
clients for extended periods of time, to prevent harm to the service. If
your client can not abide by 429/Retry-After headers, please do not run
it in automated repeated fashion until it either can handle them
properly, or insert delays long enough so you can be sure you are not
launching an avalanche of heavy requests and crowding out other users.
If something takes too long, that's a good moment to ask for help, not
to put it in a loop that would hit the server repeatedly for days.
If you need to deal with a massive data set that needs to be processed,
I would suggest trying the following strategy:
1. Load the primary key data - like list of all humans if that's what
you need - to your own storage. You can use either LDF server or parsing
the dump directly for that for Q5 (maybe with Wikidata Toolkit?). For
some scenarios, even direct query would be fine, but for Q5 it probably
would be too much.
2. Split this data set into palatable batches - like 100 items per batch
or so, you can experiment on that, it's fine to cause a couple of
timeouts if it's not an automated script doing it 20 times a second for
a long time. Once you have sane batch size, run the query that needs to
fetch other data using VALUES clause to substitute primary key data.
Watch the 429 responses - if you're getting them, insert delays or lower
batch size, or ask for help again if it doesn't work.
Alternatively, segmenting the records by some other criteria may work
too, but I don't think filter like STRSTARTS(?personLabel, "D")) is
going to be effective - I don't think Blazegraph query optimizer is
smart enough to convert this to index lookup, and without that, this is
just slowing things down by introducing more checks in the query. And
even if it did, there's a lot of labels starting with "D", so that
probably won't be too useful for speeding it up.
Having said that, I am curious - what exactly you are doing with this
data set? Why you need a list of all humans - how this list is going to
be used? Knowing that may help to devise better specialized strategy of
achieving the same.
--
Stas Malyshev
smalyshev(a)wikimedia.org