Hoi, Two things, not everybody has the capacity to run an instance of the toolkit. When there are other reasons as well for needing a toolkit than query does not cope, it makes sense to have instances of toolkit on labs where queries like this can be run. Your response is technical and seriously, query is a tool and it should function for people. When the tool is not good enough fix it. You cannot expect people to engage in the toolkit because most queries are community incidentals and not part of a scientific endeavour. Thanks, GerardM On 11 February 2016 at 01:34, Stas Malyshev <smalyshev(a)wikimedia.org> wrote:

Hoi, What I hear is that the intentions were wrong in that you did not anticipate people to get actual meaningful requests out of it. When you state "we have two choices", you imply that it is my choice as well. It is not. The answer that I am looking for is yes, it does not function as we would like, we are working on it and in the mean time we will ensure that toolkit is available on Labs for the more complex queries. Wikidata is a service and the service is in need of being better. Thanks, GerardM On 11 February 2016 at 12:32, Daniel Kinzler <daniel.kinzler(a)wikimedia.de> wrote:

Hoi,' Markus when you read my reply on the original question you will see that my approach is different. The first thing that I pointed out was that a technical assumption has little to do with what people need. I indicated that when this is the approach, the answer is fix it. The notion that a large number of returns is outrageous is not of this time. My approach was one where I even offered a possible solution, a crutch. The approach Daniel took was to make me look ridiculous. His choice, not mine. I stayed polite and told him that his answers are not my answers and why. The point that I make is that Wikidata is a service. It will increasingly be used for the most outrageous queries and people will expect it to work because why else do we put all this data in there. Why else is this the data hub for Wikipedia. Why else Do appreciate that the aim of the WMF is to share in the sum of all available knowledge. When the current technology is what we have to make do with, fine for now. Say so, but do not ridicule me for saying that it is not good enough, it is not now and it will certainly not be in the future... Thanks, GerardM On 11 February 2016 at 15:25, Markus Krötzsch &lt;markus(a)semantic-mediawiki.org > On 11.02.2016 15:01, Gerard Meijssen wrote: > >> Hoi, >> What I hear is that the intentions were wrong in that you did not >> anticipate people to get actual meaningful requests out of it. >> >> When you state "we have two choices", you imply that it is my choice as >> well. It is not. The answer that I am looking for is yes, it does not >> function as we would like, we are working on it and in the mean time we >> will ensure that toolkit is available on Labs for the more complex >> queries. >> >> Wikidata is a service and the service is in need of being better. >> > > Gerard, do you realise how far away from technical reality your wishes > are? We are far ahead of the state of the art in what we already have for > Wikidata: two powerful live query services + a free toolkit for batch > analyses + several Web APIs for live lookups. I know of no site of this > scale that is anywhere near this in terms of functionality. You can always > ask for more, but you should be a bit reasonable too, or people will just > ignore you. > > Markus > > > On 11 February 2016 at 12:32, Daniel Kinzler >> &lt;daniel.kinzler(a)wikimedia.de <mailto:daniel.kinzler@wikimedia.de>>> >> Am 11.02.2016 um 10:17 schrieb Gerard Meijssen: >> > Your response is technical and seriously, query is a tool and it >> should function >> > for people. When the tool is not good enough fix it. >> >> What I hear: "A hammer is a tool, it should work for people. Tearing >> down a >> building with it takes forever, so fix the hammer!" >> >> The query service was never intended to run arbitrarily large or >> complex >> queries. Sure, would be nice, but that also means committing an >> arbitrary amount >> of resources to a single request. We don't have arbitrary amounts of >> resources. >> >> We basically have two choices: either we offer a limited interface >> that only >> allows for a narrow range of queries to be run at all. Or we offer a >> very >> general interface that can run arbitrary queries, but we impose >> limits on time >> and memory consumption. I would actually prefer the first option, >> because it's >> more predictable, and doesn't get people's hopes up too far. What do >> you think? >> >> Oh, and +1 for making it easy to use WDT on labs. >> >> -- >> Daniel Kinzler >> Senior Software Developer >> >> Wikimedia Deutschland >> Gesellschaft zur Förderung Freien Wissens e.V. >> >> _______________________________________________ >> Wikidata mailing list >> Wikidata(a)lists.wikimedia.org <mailto:Wikidata@lists.wikimedia.org> >> https://lists.wikimedia.org/mailman/listinfo/wikidata >> >> >> >> >> _______________________________________________ >> Wikidata mailing list >> Wikidata(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/wikidata >> >> > > _______________________________________________ > Wikidata mailing list > Wikidata(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikidata >

Hi Lydia, I agree on using the right tool for the job. Yet, it isn’t always obvious what is right and what the limitations of a tool are. For me, it’s perfectly ok when a query runs for 20 minutes, when it spares me some hours of setting up a specific environment for one specific dataset (and doing it again when I need current data two month later). And it would be no issue if the query runs much longer, in situations where it competes with several others. But of course, that’s not what I want to experience when I use a wikidata service to drive, e.g., an autosuggest function for selecting entities. So, can you agree to Markus suggestion that an experimental “unstable” endpoint could solve different use cases and expectiations? And do you think the policies and limitations of different access strategies could be documented? These could include a high-reliability interface for a narrow range of queries (as Daniel suggests as his preferred option). And on the other end of the spectrum something what allows people to experiment freely. Finally, the latter kind of interface could allow new patterns of usage to evolve, with perhaps a few of them worthwhile to become part of an optimized, highly reliabile query set. I could imagine that such a documentation of (and perhaps discussion on) different options and access strategies, limitations and tradeoffs could solve Gerards claim to give people what they need, or at least let them make informed choises when restrictions are unavoidable. Cheers, Joachim Von: Wikidata [mailto:wikidata-bounces@lists.wikimedia.org] Im Auftrag von Lydia Pintscher Gesendet: Donnerstag, 11. Februar 2016 17:55 An: Discussion list for the Wikidata project. Betreff: Re: [Wikidata] SPARQL CONSTRUCT results truncated On Thu, Feb 11, 2016 at 5:53 PM Gerard Meijssen <gerard.meijssen@gmail.com<mailto:gerard.meijssen@gmail.com>> wrote: Hoi,' Markus when you read my reply on the original question you will see that my approach is different. The first thing that I pointed out was that a technical assumption has little to do with what people need. I indicated that when this is the approach, the answer is fix it. The notion that a large number of returns is outrageous is not of this time. My approach was one where I even offered a possible solution, a crutch. The approach Daniel took was to make me look ridiculous. His choice, not mine. I stayed polite and told him that his answers are not my answers and why. The point that I make is that Wikidata is a service. It will increasingly be used for the most outrageous queries and people will expect it to work because why else do we put all this data in there. Why else is this the data hub for Wikipedia. Why else Do appreciate that the aim of the WMF is to share in the sum of all available knowledge. When the current technology is what we have to make do with, fine for now. Say so, but do not ridicule me for saying that it is not good enough, it is not now and it will certainly not be in the future... Thanks, GerardM Gerard, it all boils down to using the right tool for the job. Nothing more - nothing less. Let's get back to making Wikidata rock. Cheers Lydia -- Lydia Pintscher - http://about.me/lydia.pintscher Product Manager for Wikidata Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de<http://www.wikimedia.de> Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

Hi! I understand that, but this is a shared server which is supposed to serve many users, and if we allow to run 20-minute queries on this service, soon enough it would become unusable. This is why we have 30-second limit on the server. Now, we have considered having an option for the server or setup that allows to run longer queries, but currently we don't have one. It would require some budget allocation and work to make it, so it's not something we can have right now. There are use cases for very long queries and very large results, the current public service endpoint is just not good in serving them, because it's not what it was meant for. I agree that limitations better to be documented, the problem is we don't know everything we may need to document. Such as "what are queries that may be bad". When I see something like "I want to download million-row dataset" I know it's probably a bit too much. But I can't have hard rule that says 1M-1 is ok, but 1M is too much. I'm not sure how I could maintain an endpoint that would allow people to do anything they want and still provide adequate experience for everybody. Maybe if we had infinite hardware resources... but we do not. Otherwise, it is possible - and should not be extremely hard - to set one's own instance of the Query Service and use it for experimenting with heavy lifting. Of course, that would require resources - but there's no magic here, it'd require resources from us too, both in terms of hardware and people that would maintain it. So some things we can do now, some things we would be able to do later, and some things we probably would not be able to offer with any adequate quality. -- Stas Malyshev smalyshev(a)wikimedia.org

On 13.02.2016 23:50, Kingsley Idehen wrote: ... I think this does not help here. If I only ask for part of the data (see my previous email), I can get all 300K results in 9.3sec. The size of the result does not seem to be the issue. If I add further joins to the query, the time needed seems to go above 10sec (timeout) even with a LIMIT. Note that you need to order results for using LIMIT in a reliable way, since the data changes by the minute and the "natural" order of results would change as well. I guess with a blocking operator like ORDER BY in the equation, the use of LIMIT does not really save much time (other than for final result serialisation and transfer, which seems pretty quick). Markus -- Markus Kroetzsch Faculty of Computer Science Technische Universität Dresden +49 351 463 38486 http://korrekt.org/

Hi! That would require implementing pretty smart SPARQL parser... I don't think it worth the investment of time. I'd rather put caps on runtime and maybe also on parallel queries per IP, to ensure fair access. We may also have a way to run longer queries - in fact, we'll need it anyway if we want to automate lists - but that is longer term, we'll need to figure out infrastructure for that and how we allocate access. -- Stas Malyshev smalyshev(a)wikimedia.org

On 12.02.2016 00:04, Stas Malyshev wrote: +1 Restricting queries syntactically to be "simpler" is what we did in Semantic MediaWiki (because MySQL did not support time/memory limits per query). It is a workaround, but it will not prevent long-running queries unless you make the syntactic restrictions really severe (and thereby forbid many simple queries, too). I would not do it if there is support for time/memory limits instead. In the end, even the SPARQL engines are not able to predict reliably how complicated a query is going to be -- it's an important part of their work (for optimising query execution), but it is also very difficult. Markus

On 12.02.2016 10:01, Osma Suominen wrote: There already is such a service for Wikidata (Cristian Consonni has set it up a while ago). You could try if the query would work there. I think that such queries would be rather challenging for a server of this type, since they require you to iterate almost all of the data client-side. Note that both "instance of human" and "has a GND identifier" are not very selective properties. In this sense, the queries may not be "relatively easy to serve" in this particular case. Markus

Hi! There is apparently this: https://github.com/CristianCantoro/wikidataldf though not sure what it its status - I just found it. In general, yes, I think checking out LDF may be a good idea. I'll put it on my todo list. -- Stas Malyshev smalyshev(a)wikimedia.org

And here is another comment on this interesting topic :-) I just realised how close the service is to answering the query. It turns out that you can in fact get the whole set of (currently >324000 result items) together with their GND identifiers as a download *within the timeout* (I tried several times without any errors). This is a 63M json result file with >640K individual values, and it downloads in no time on my home network. The query I use is simply this: PREFIX wd: <http://www.wikidata.org/entity/> PREFIX wdt: <http://www.wikidata.org/prop/direct/> select ?item ?gndId where { ?item wdt:P227 ?gndId ; # get gnd ID wdt:P31 wd:Q5 . # instance of human } ORDER BY ASC(?gndId) LIMIT 10 (don't run this in vain: even with the limit, the ORDER clause requires the service to compute all results every time someone runs this. Also be careful when removing the limit; your browser may hang on an HTML page that large; better use the SPARQL endpoint directly to download the complete result file.) It seems that the timeout is only hit when adding more information (labels and wiki URLs) to the result. So it seems that we are not actually very far away from being able to answer the original query even within the timeout. Certainly not as far away as I first thought. It might not be necessary at all to switch to a different approach (though it would be interesting to know how long LDF takes to answer the above -- our current service takes less than 10sec). Cheers, Markus On 13.02.2016 11:40, Peter Haase wrote: -- Markus Kroetzsch Faculty of Computer Science Technische Universität Dresden +49 351 463 38486 http://korrekt.org/

Hi! That's a very good point. We currently do not have robots.txt file on the service. We should have it. I'll fix it ASAP. GUI links do not run the query until click, so they are safe from bots anyway. But direct links to sparql endpoint do run the query (it's the API after all :) So robots.txt is needed there. -- Stas Malyshev smalyshev(a)wikimedia.org

On 13.02.2016 23:56, Kingsley Idehen wrote: ... To clarify: I just added the LIMIT to prevent unwary readers from killing their browser on a 100MB HTML result page. The server does not need it at all and can give you all result at once. Online applications may still want to scroll results, I agree, but for the OP it would be more useful to just donwload one file here. Markus -- Markus Kroetzsch Faculty of Computer Science Technische Universität Dresden +49 351 463 38486 http://korrekt.org/

Hi Markus, Great that you checked that out. I can confirm that the simplified query worked for me, too. It took 15.6s and revealed roughly the same number of results (323789). When I loaded the results into http://zbw.eu/beta/sparql/econ_pers/query, an endpoint for "economics-related" persons, it matched with 36050 persons (supposedly the "most important" 8 percent of our set). What I normally would do to get the according Wikipedia site URLs, is a query against the wikidata endpoint, which references the relevant wikidata URIs via a "service" clause: PREFIX skos: <http://www.w3.org/2004/02/skos/core#> PREFIX schema: <http://schema.org/> # construct { ?gnd schema:about ?sitelink . } where { service <http://zbw.eu/beta/sparql/econ_pers/query> { ?gnd skos:prefLabel [] ; skos:exactMatch ?wd . filter(contains(str(?wd), 'wikidata')) } ?sitelink schema:about ?wd ; schema:inLanguage ?language . filter (contains(str(?sitelink), 'wikipedia')) filter (lang(?wdLabel) = ?language && ?language in ('en', 'de')) } This however results in a java error. If "service" clauses are supposed to work in the wikidata endpoint, I'd happily provide addtitional details in phabricator. For now, I'll get the data via your java example code :) Cheers, Joachim -----Ursprüngliche Nachricht----- Von: Wikidata [mailto:wikidata-bounces@lists.wikimedia.org] Im Auftrag von Markus Kroetzsch Gesendet: Samstag, 13. Februar 2016 22:56 An: Discussion list for the Wikidata project. Betreff: Re: [Wikidata] SPARQL CONSTRUCT results truncated And here is another comment on this interesting topic :-) I just realised how close the service is to answering the query. It turns out that you can in fact get the whole set of (currently >324000 result items) together with their GND identifiers as a download *within the timeout* (I tried several times without any errors). This is a 63M json result file with >640K individual values, and it downloads in no time on my home network. The query I use is simply this: PREFIX wd: <http://www.wikidata.org/entity/> PREFIX wdt: <http://www.wikidata.org/prop/direct/> select ?item ?gndId where { ?item wdt:P227 ?gndId ; # get gnd ID wdt:P31 wd:Q5 . # instance of human } ORDER BY ASC(?gndId) LIMIT 10 (don't run this in vain: even with the limit, the ORDER clause requires the service to compute all results every time someone runs this. Also be careful when removing the limit; your browser may hang on an HTML page that large; better use the SPARQL endpoint directly to download the complete result file.) It seems that the timeout is only hit when adding more information (labels and wiki URLs) to the result. So it seems that we are not actually very far away from being able to answer the original query even within the timeout. Certainly not as far away as I first thought. It might not be necessary at all to switch to a different approach (though it would be interesting to know how long LDF takes to answer the above -- our current service takes less than 10sec). Cheers, Markus On 13.02.2016 11:40, Peter Haase wrote: -- Markus Kroetzsch Faculty of Computer Science Technische Universität Dresden +49 351 463 38486 http://korrekt.org/ _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Thanks Markus, I've created https://phabricator.wikimedia.org/T127070 with the details. -----Ursprüngliche Nachricht----- Von: Wikidata [mailto:wikidata-bounces@lists.wikimedia.org] Im Auftrag von Markus Krötzsch Gesendet: Dienstag, 16. Februar 2016 14:57 An: Discussion list for the Wikidata project. Betreff: Re: [Wikidata] SPARQL CONSTRUCT results truncated Hi Joachim, I think SERVICE queries should be working, but maybe Stas knows more about this. Even if they are disabled, this should not result in some message rather than in a NullPointerException. Looks like a bug. Markus On 16.02.2016 13:56, Neubert, Joachim wrote: _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Hi Joachim, Here is a short program that solves your problem: https://github.com/Wikidata/Wikidata-Toolkit-Examples/blob/master/src/examp… It is in Java, so, you need that (and Maven) to run it, but that's the only technical challenge ;-). You can run the program in various ways as described in the README: https://github.com/Wikidata/Wikidata-Toolkit-Examples The program I wrote puts everything into a CSV file, but you can of course also write RDF triples if you prefer this, or any other format you wish. The code should be easy to modify. On a first run, the tool will download the current Wikidata dump, which takes a while (it's about 6G), but after this you can find and serialise all results in less than half an hour (for a processing rate of around 10K items/second). A regular laptop is enough to run it. Cheers, Markus On 11.02.2016 01:34, Stas Malyshev wrote:

Hi Stas, Thanks for your answer. You asked how long the query runs: 8.21 sec (having processed 6443 triples), in an example invocation. If roughly linear, that could mean 800-1500 sec for the whole set. However, I would expect a clearly shorter runtime: I routinely use queries of similar complexity and result sizes on ZBW's public endpoints. One arbitrary selected query which extracts data from GND runs for less than two minutes to produce 1.2m triples. Given the size of Wikidata, I wouldn't consider such an use abusive. Of course, if you have lots of competing queries and resources are limited, it is completely legitimate to implement some policy which formulates limits and enforces them technically (throddle down long-running queries, or limit the number of produced triples, or the execution time, or whatever seems reasonable and can be implemented). Anyway, in this case (truncation in the middle of a statement), it looks much more like some technical bug (or an obscure timeout somewhere down the way). The execution time and the result size varies widely: 5.44s empty result 8.60s 2090 triples 5.44s empty result 22.70s 27352 triples Can you reproduce this kind of results with the given query, or with other supposedly longer-running queries? Thanks again for looking into this. Cheers, Joachim PS. I plan to set up an own Wikidata SPAQL endpoint to do more complex things, but that depends on a new machine which will be available in some month. For now, I'd just like to know which for "our" persons (economists and the like) have wikipedia pages. PPS. From my side, I would much more have liked to build a query which asks for exactly the GND IDs I'm interested in (about 430.000 out of millions of GNDs). This would have led to a much smaller result - but I cannot squeeze that query into a GET request ... -----Ursprüngliche Nachricht----- Von: Wikidata [mailto:wikidata-bounces@lists.wikimedia.org] Im Auftrag von Stas Malyshev Gesendet: Donnerstag, 11. Februar 2016 01:35 An: Discussion list for the Wikidata project. Betreff: Re: [Wikidata] SPARQL CONSTRUCT results truncated Hi! As a starting note, I don't think extracting 1M triples may be the best way to use query service. If you need to do processing that returns such big result sets - in millions - maybe processing the dump - e.g. with wikidata toolkit at https://github.com/Wikidata/Wikidata-Toolkit - would be better idea? It may be some kind of timeout because of the quantity of the data being sent. How long does such request take? -- Stas Malyshev smalyshev(a)wikimedia.org _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Hi! That looks weirdly random. I'll check out what is going on there. -- Stas Malyshev smalyshev(a)wikimedia.org