Re: [Wikidata] WDQS with use of automated requests

Hi! On 5/15/18 3:27 PM, Justin Maltais wrote: The first thing I would like to ask is please don't do that again. This created a significant load on the server, the script completely ignored the throttling headers we sent, and in the future we would ban such clients for extended periods of time, to prevent harm to the service. If your client can not abide by 429/Retry-After headers, please do not run it in automated repeated fashion until it either can handle them properly, or insert delays long enough so you can be sure you are not launching an avalanche of heavy requests and crowding out other users. If something takes too long, that's a good moment to ask for help, not to put it in a loop that would hit the server repeatedly for days. If you need to deal with a massive data set that needs to be processed, I would suggest trying the following strategy: 1. Load the primary key data - like list of all humans if that's what you need - to your own storage. You can use either LDF server or parsing the dump directly for that for Q5 (maybe with Wikidata Toolkit?). For some scenarios, even direct query would be fine, but for Q5 it probably would be too much. 2. Split this data set into palatable batches - like 100 items per batch or so, you can experiment on that, it's fine to cause a couple of timeouts if it's not an automated script doing it 20 times a second for a long time. Once you have sane batch size, run the query that needs to fetch other data using VALUES clause to substitute primary key data. Watch the 429 responses - if you're getting them, insert delays or lower batch size, or ask for help again if it doesn't work. Alternatively, segmenting the records by some other criteria may work too, but I don't think filter like STRSTARTS(?personLabel, "D")) is going to be effective - I don't think Blazegraph query optimizer is smart enough to convert this to index lookup, and without that, this is just slowing things down by introducing more checks in the query. And even if it did, there's a lot of labels starting with "D", so that probably won't be too useful for speeding it up. Having said that, I am curious - what exactly you are doing with this data set? Why you need a list of all humans - how this list is going to be used? Knowing that may help to devise better specialized strategy of achieving the same. -- Stas Malyshev smalyshev(a)wikimedia.org