Re: [Wikidata] Can LDF scale?

Hi Stas, Ruben, Thad, Collected answers: * Bandwidth: I think this is not the main issue. The slowness is a bonus here, since it stretches the time during which the actual data is loaded. I am not sure but would guess that my 1h20min query has not received much more than 100MB of data. * Caching: Ruben's arguments about this being a "finite set of URLs to cache" is misleading, and, I believe, not true. Here is why: (1) It is not true since we need to cache triple patterns taken from queries, not patterns from the data. For example, there are infinitely many different strings and for each you could look for all Wikidata entities with this label (in fact, this is a real use case that is quite common for WDQS, since people check for labels that are not in the data when doing database integration). Even if there are zero results, this must either be computed or returned from cache. (2) It is misleading since the number of triple patterns is astronomic even when restricting to URLs and literals that exist in Wikidata: -> Number of Item-Property-Item triple patterns with one variable: 23M x 829 x 2 + 23M x 23M = 529 * 10^12 (0.5 Quadrillion) (829 is the current number of item properties in Wikidata) Each pattern may lead to many URLs, since the results are split into pages if there are many. The total number is therefore slightly bigger (not much, since most of the patterns have no results -- but this still needs to be cached!). So the situation of "what can I cache in practice" vs. "what would I need to cache in theory" is the same as in SPARQL: there is far too much to cache to ever get anywhere near completeness. * Can caching still help? In a real-world setting, you would like to update your data once in a while. The real question for caching is which percentage of your requests are served from cache *during this time*. In the calculation I made, there were around 100K requests per minute. If we take this as the interval between data updates (as in WDS now), then chances seem pretty slim that a request repeats. One could hope that there are only few requests and that they therefore repeat a lot. My test suggests that this is not so likely, since my single query generated hundred thousands of requests, which I assume were all different (why fetch the same data twice in one query execution?). In any case, the relative benefit of caching will surely be much smaller for TPF (where you cache an easy-to-compute but large answer) than for SPARQL endpoints (where you cache a hard-to-compute but often small answer). Also note that the total number of SPARQL queries one needs to cache is much (much!) smaller than the number of TPF replies one would cache for answering them. * Are request numbers no problem because requests are simple? This really depends on the database implementation behind the service. Surely it helps if the database has less work. But there is also the notion of "thrashing" a database, where you destroy performance not by difficult queries but by large numbers of simple ones. Moreover, other components of the server could also be challenged, such as logging or other types of filtering that happen before you even get to the DB. I realise that I am being rather critical here, but I think we must cross-check if it is realistic to assume that this approach could become useful for Wikidata at some point before investing much more staff time into it. I have supported this work as a research work, and I still do, but the impression we got so far in the "wild" is really not very encouraging. Best regards, Markus On 22.12.2016 18:30, Stas Malyshev wrote: