Hoi, Two things, not everybody has the capacity to run an instance of the toolkit. When there are other reasons as well for needing a toolkit than query does not cope, it makes sense to have instances of toolkit on labs where queries like this can be run. Your response is technical and seriously, query is a tool and it should function for people. When the tool is not good enough fix it. You cannot expect people to engage in the toolkit because most queries are community incidentals and not part of a scientific endeavour. Thanks, GerardM On 11 February 2016 at 01:34, Stas Malyshev <smalyshev(a)wikimedia.org> wrote:

Hoi, What I hear is that the intentions were wrong in that you did not anticipate people to get actual meaningful requests out of it. When you state "we have two choices", you imply that it is my choice as well. It is not. The answer that I am looking for is yes, it does not function as we would like, we are working on it and in the mean time we will ensure that toolkit is available on Labs for the more complex queries. Wikidata is a service and the service is in need of being better. Thanks, GerardM On 11 February 2016 at 12:32, Daniel Kinzler <daniel.kinzler(a)wikimedia.de> wrote:

On 11.02.2016 15:01, Gerard Meijssen wrote: Gerard, do you realise how far away from technical reality your wishes are? We are far ahead of the state of the art in what we already have for Wikidata: two powerful live query services + a free toolkit for batch analyses + several Web APIs for live lookups. I know of no site of this scale that is anywhere near this in terms of functionality. You can always ask for more, but you should be a bit reasonable too, or people will just ignore you. Markus

Hi Joachim, I think the problem is not to answer your query in 5min or so (Wikidata Toolkit on my laptop takes 27min without a database, by simply parsing the whole data file, so any database that already has the data should be much faster). The bigger issue is that you would have to configure the site to run for 5min before timeout. This would mean that other queries that never terminate (because they are really hard) also can run for at least this time. It seems that this could easily cause the service to break down. Maybe one could have an "unstable" service on a separate machine that does the same as WDQS but with a much more liberal timeout and less availability (if it's overloaded a lot, it will just be down more often, but you would know when you use it that this is the deal). Cheers, Markus On 11.02.2016 15:54, Neubert, Joachim wrote:

Hoi,' Markus when you read my reply on the original question you will see that my approach is different. The first thing that I pointed out was that a technical assumption has little to do with what people need. I indicated that when this is the approach, the answer is fix it. The notion that a large number of returns is outrageous is not of this time. My approach was one where I even offered a possible solution, a crutch. The approach Daniel took was to make me look ridiculous. His choice, not mine. I stayed polite and told him that his answers are not my answers and why. The point that I make is that Wikidata is a service. It will increasingly be used for the most outrageous queries and people will expect it to work because why else do we put all this data in there. Why else is this the data hub for Wikipedia. Why else Do appreciate that the aim of the WMF is to share in the sum of all available knowledge. When the current technology is what we have to make do with, fine for now. Say so, but do not ridicule me for saying that it is not good enough, it is not now and it will certainly not be in the future... Thanks, GerardM On 11 February 2016 at 15:25, Markus Krötzsch &lt;markus(a)semantic-mediawiki.org > On 11.02.2016 15:01, Gerard Meijssen wrote: > >> Hoi, >> What I hear is that the intentions were wrong in that you did not >> anticipate people to get actual meaningful requests out of it. >> >> When you state "we have two choices", you imply that it is my choice as >> well. It is not. The answer that I am looking for is yes, it does not >> function as we would like, we are working on it and in the mean time we >> will ensure that toolkit is available on Labs for the more complex >> queries. >> >> Wikidata is a service and the service is in need of being better. >> > > Gerard, do you realise how far away from technical reality your wishes > are? We are far ahead of the state of the art in what we already have for > Wikidata: two powerful live query services + a free toolkit for batch > analyses + several Web APIs for live lookups. I know of no site of this > scale that is anywhere near this in terms of functionality. You can always > ask for more, but you should be a bit reasonable too, or people will just > ignore you. > > Markus > > > On 11 February 2016 at 12:32, Daniel Kinzler >> &lt;daniel.kinzler(a)wikimedia.de <mailto:daniel.kinzler@wikimedia.de>>> >> Am 11.02.2016 um 10:17 schrieb Gerard Meijssen: >> > Your response is technical and seriously, query is a tool and it >> should function >> > for people. When the tool is not good enough fix it. >> >> What I hear: "A hammer is a tool, it should work for people. Tearing >> down a >> building with it takes forever, so fix the hammer!" >> >> The query service was never intended to run arbitrarily large or >> complex >> queries. Sure, would be nice, but that also means committing an >> arbitrary amount >> of resources to a single request. We don't have arbitrary amounts of >> resources. >> >> We basically have two choices: either we offer a limited interface >> that only >> allows for a narrow range of queries to be run at all. Or we offer a >> very >> general interface that can run arbitrary queries, but we impose >> limits on time >> and memory consumption. I would actually prefer the first option, >> because it's >> more predictable, and doesn't get people's hopes up too far. What do >> you think? >> >> Oh, and +1 for making it easy to use WDT on labs. >> >> -- >> Daniel Kinzler >> Senior Software Developer >> >> Wikimedia Deutschland >> Gesellschaft zur Förderung Freien Wissens e.V. >> >> _______________________________________________ >> Wikidata mailing list >> Wikidata(a)lists.wikimedia.org <mailto:Wikidata@lists.wikimedia.org> >> https://lists.wikimedia.org/mailman/listinfo/wikidata >> >> >> >> >> _______________________________________________ >> Wikidata mailing list >> Wikidata(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/wikidata >> >> > > _______________________________________________ > Wikidata mailing list > Wikidata(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikidata >

Hi Lydia, I agree on using the right tool for the job. Yet, it isn’t always obvious what is right and what the limitations of a tool are. For me, it’s perfectly ok when a query runs for 20 minutes, when it spares me some hours of setting up a specific environment for one specific dataset (and doing it again when I need current data two month later). And it would be no issue if the query runs much longer, in situations where it competes with several others. But of course, that’s not what I want to experience when I use a wikidata service to drive, e.g., an autosuggest function for selecting entities. So, can you agree to Markus suggestion that an experimental “unstable” endpoint could solve different use cases and expectiations? And do you think the policies and limitations of different access strategies could be documented? These could include a high-reliability interface for a narrow range of queries (as Daniel suggests as his preferred option). And on the other end of the spectrum something what allows people to experiment freely. Finally, the latter kind of interface could allow new patterns of usage to evolve, with perhaps a few of them worthwhile to become part of an optimized, highly reliabile query set. I could imagine that such a documentation of (and perhaps discussion on) different options and access strategies, limitations and tradeoffs could solve Gerards claim to give people what they need, or at least let them make informed choises when restrictions are unavoidable. Cheers, Joachim Von: Wikidata [mailto:wikidata-bounces@lists.wikimedia.org] Im Auftrag von Lydia Pintscher Gesendet: Donnerstag, 11. Februar 2016 17:55 An: Discussion list for the Wikidata project. Betreff: Re: [Wikidata] SPARQL CONSTRUCT results truncated On Thu, Feb 11, 2016 at 5:53 PM Gerard Meijssen <gerard.meijssen@gmail.com<mailto:gerard.meijssen@gmail.com>> wrote: Hoi,' Markus when you read my reply on the original question you will see that my approach is different. The first thing that I pointed out was that a technical assumption has little to do with what people need. I indicated that when this is the approach, the answer is fix it. The notion that a large number of returns is outrageous is not of this time. My approach was one where I even offered a possible solution, a crutch. The approach Daniel took was to make me look ridiculous. His choice, not mine. I stayed polite and told him that his answers are not my answers and why. The point that I make is that Wikidata is a service. It will increasingly be used for the most outrageous queries and people will expect it to work because why else do we put all this data in there. Why else is this the data hub for Wikipedia. Why else Do appreciate that the aim of the WMF is to share in the sum of all available knowledge. When the current technology is what we have to make do with, fine for now. Say so, but do not ridicule me for saying that it is not good enough, it is not now and it will certainly not be in the future... Thanks, GerardM Gerard, it all boils down to using the right tool for the job. Nothing more - nothing less. Let's get back to making Wikidata rock. Cheers Lydia -- Lydia Pintscher - http://about.me/lydia.pintscher Product Manager for Wikidata Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de<http://www.wikimedia.de> Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

Hi! That looks weirdly random. I'll check out what is going on there. -- Stas Malyshev smalyshev(a)wikimedia.org

Hoi, This is the kind of (technical) feedback that makes sense as it is centred on need. It acknowledges that more needs to be done as we are not ready for what we expect of ourselves in the first place. In this day and age of big data, we are a very public place where a lot of initiatives gravitate to. If the WMF wants to retain its relevance, it is to face its challenges. Maybe the WDQS can steal a page out of the architecture in what Magnus build. It is very much replicable and multiple instances have been running. This is not to say that it becomes more and more relevant to have the Wikidata toolkit available from Labs with as many instances as needed. Thanks, GerardM On 12 February 2016 at 00:04, Stas Malyshev &lt;smalyshev(a)wikimedia.org&gt; wrote:

On 12.02.2016 00:04, Stas Malyshev wrote: +1 Restricting queries syntactically to be "simpler" is what we did in Semantic MediaWiki (because MySQL did not support time/memory limits per query). It is a workaround, but it will not prevent long-running queries unless you make the syntactic restrictions really severe (and thereby forbid many simple queries, too). I would not do it if there is support for time/memory limits instead. In the end, even the SPARQL engines are not able to predict reliably how complicated a query is going to be -- it's an important part of their work (for optimising query execution), but it is also very difficult. Markus

On 12.02.2016 10:01, Osma Suominen wrote: There already is such a service for Wikidata (Cristian Consonni has set it up a while ago). You could try if the query would work there. I think that such queries would be rather challenging for a server of this type, since they require you to iterate almost all of the data client-side. Note that both "instance of human" and "has a GND identifier" are not very selective properties. In this sense, the queries may not be "relatively easy to serve" in this particular case. Markus

Hi! There is apparently this: https://github.com/CristianCantoro/wikidataldf though not sure what it its status - I just found it. In general, yes, I think checking out LDF may be a good idea. I'll put it on my todo list. -- Stas Malyshev smalyshev(a)wikimedia.org

And here is another comment on this interesting topic :-) I just realised how close the service is to answering the query. It turns out that you can in fact get the whole set of (currently >324000 result items) together with their GND identifiers as a download *within the timeout* (I tried several times without any errors). This is a 63M json result file with >640K individual values, and it downloads in no time on my home network. The query I use is simply this: PREFIX wd: <http://www.wikidata.org/entity/> PREFIX wdt: <http://www.wikidata.org/prop/direct/> select ?item ?gndId where { ?item wdt:P227 ?gndId ; # get gnd ID wdt:P31 wd:Q5 . # instance of human } ORDER BY ASC(?gndId) LIMIT 10 (don't run this in vain: even with the limit, the ORDER clause requires the service to compute all results every time someone runs this. Also be careful when removing the limit; your browser may hang on an HTML page that large; better use the SPARQL endpoint directly to download the complete result file.) It seems that the timeout is only hit when adding more information (labels and wiki URLs) to the result. So it seems that we are not actually very far away from being able to answer the original query even within the timeout. Certainly not as far away as I first thought. It might not be necessary at all to switch to a different approach (though it would be interesting to know how long LDF takes to answer the above -- our current service takes less than 10sec). Cheers, Markus On 13.02.2016 11:40, Peter Haase wrote: -- Markus Kroetzsch Faculty of Computer Science Technische Universität Dresden +49 351 463 38486 http://korrekt.org/

On 2/11/16 9:25 AM, Markus Krötzsch wrote: Markus and others interested in this matter, What about using OFFSET and LIMIT to address this problem? That's what we advice users of the DBpedia endpoint (and others we publish) to do. We have to educate people about query implications and options. Even after that, you have the issue of timeouts (which aren't part of the SPARQL spec) that can be used to produce partial results (notified via HTTP headers), but that's something that comes after the basic scrolling functionality of OFFSET and LIMIT are understood. [1] http://stackoverflow.com/questions/20937556/how-to-get-all-companies-from-d… [2] https://sourceforge.net/p/dbpedia/mailman/message/29172307/ -- Regards, Kingsley Idehen Founder & CEO OpenLink Software Company Web: http://www.openlinksw.com Personal Weblog 1: http://kidehen.blogspot.com Personal Weblog 2: http://www.openlinksw.com/blog/~kidehen Twitter Profile: https://twitter.com/kidehen Google+ Profile: https://plus.google.com/+KingsleyIdehen/about LinkedIn Profile: http://www.linkedin.com/in/kidehen Personal WebID: http://kingsley.idehen.net/dataspace/person/kidehen#this

On 13.02.2016 23:50, Kingsley Idehen wrote: ... I think this does not help here. If I only ask for part of the data (see my previous email), I can get all 300K results in 9.3sec. The size of the result does not seem to be the issue. If I add further joins to the query, the time needed seems to go above 10sec (timeout) even with a LIMIT. Note that you need to order results for using LIMIT in a reliable way, since the data changes by the minute and the "natural" order of results would change as well. I guess with a blocking operator like ORDER BY in the equation, the use of LIMIT does not really save much time (other than for final result serialisation and transfer, which seems pretty quick). Markus -- Markus Kroetzsch Faculty of Computer Science Technische Universität Dresden +49 351 463 38486 http://korrekt.org/

Hi! That's a very good point. We currently do not have robots.txt file on the service. We should have it. I'll fix it ASAP. GUI links do not run the query until click, so they are safe from bots anyway. But direct links to sparql endpoint do run the query (it's the API after all :) So robots.txt is needed there. -- Stas Malyshev smalyshev(a)wikimedia.org

Hi Markus, Great that you checked that out. I can confirm that the simplified query worked for me, too. It took 15.6s and revealed roughly the same number of results (323789). When I loaded the results into http://zbw.eu/beta/sparql/econ_pers/query, an endpoint for "economics-related" persons, it matched with 36050 persons (supposedly the "most important" 8 percent of our set). What I normally would do to get the according Wikipedia site URLs, is a query against the wikidata endpoint, which references the relevant wikidata URIs via a "service" clause: PREFIX skos: <http://www.w3.org/2004/02/skos/core#> PREFIX schema: <http://schema.org/> # construct { ?gnd schema:about ?sitelink . } where { service <http://zbw.eu/beta/sparql/econ_pers/query> { ?gnd skos:prefLabel [] ; skos:exactMatch ?wd . filter(contains(str(?wd), 'wikidata')) } ?sitelink schema:about ?wd ; schema:inLanguage ?language . filter (contains(str(?sitelink), 'wikipedia')) filter (lang(?wdLabel) = ?language && ?language in ('en', 'de')) } This however results in a java error. If "service" clauses are supposed to work in the wikidata endpoint, I'd happily provide addtitional details in phabricator. For now, I'll get the data via your java example code :) Cheers, Joachim -----Ursprüngliche Nachricht----- Von: Wikidata [mailto:wikidata-bounces@lists.wikimedia.org] Im Auftrag von Markus Kroetzsch Gesendet: Samstag, 13. Februar 2016 22:56 An: Discussion list for the Wikidata project. Betreff: Re: [Wikidata] SPARQL CONSTRUCT results truncated And here is another comment on this interesting topic :-) I just realised how close the service is to answering the query. It turns out that you can in fact get the whole set of (currently >324000 result items) together with their GND identifiers as a download *within the timeout* (I tried several times without any errors). This is a 63M json result file with >640K individual values, and it downloads in no time on my home network. The query I use is simply this: PREFIX wd: <http://www.wikidata.org/entity/> PREFIX wdt: <http://www.wikidata.org/prop/direct/> select ?item ?gndId where { ?item wdt:P227 ?gndId ; # get gnd ID wdt:P31 wd:Q5 . # instance of human } ORDER BY ASC(?gndId) LIMIT 10 (don't run this in vain: even with the limit, the ORDER clause requires the service to compute all results every time someone runs this. Also be careful when removing the limit; your browser may hang on an HTML page that large; better use the SPARQL endpoint directly to download the complete result file.) It seems that the timeout is only hit when adding more information (labels and wiki URLs) to the result. So it seems that we are not actually very far away from being able to answer the original query even within the timeout. Certainly not as far away as I first thought. It might not be necessary at all to switch to a different approach (though it would be interesting to know how long LDF takes to answer the above -- our current service takes less than 10sec). Cheers, Markus On 13.02.2016 11:40, Peter Haase wrote: -- Markus Kroetzsch Faculty of Computer Science Technische Universität Dresden +49 351 463 38486 http://korrekt.org/ _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Thanks Markus, I've created https://phabricator.wikimedia.org/T127070 with the details. -----Ursprüngliche Nachricht----- Von: Wikidata [mailto:wikidata-bounces@lists.wikimedia.org] Im Auftrag von Markus Krötzsch Gesendet: Dienstag, 16. Februar 2016 14:57 An: Discussion list for the Wikidata project. Betreff: Re: [Wikidata] SPARQL CONSTRUCT results truncated Hi Joachim, I think SERVICE queries should be working, but maybe Stas knows more about this. Even if they are disabled, this should not result in some message rather than in a NullPointerException. Looks like a bug. Markus On 16.02.2016 13:56, Neubert, Joachim wrote: _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

On 2/13/16 6:29 PM, Markus Kroetzsch wrote: Scrolling or Paging through query solutions is a technique beneficial to clients and servers. Understanding the concept has to be part of the narrative for working with SPARQL query solutions. This is about flexibility via usage of the full functionality of SPARQL as most developers and users simply execute queries without factoring in these techniques or the impact of their queries on other system users etc.. -- Regards, Kingsley Idehen Founder & CEO OpenLink Software Company Web: http://www.openlinksw.com Personal Weblog 1: http://kidehen.blogspot.com Personal Weblog 2: http://www.openlinksw.com/blog/~kidehen Twitter Profile: https://twitter.com/kidehen Google+ Profile: https://plus.google.com/+KingsleyIdehen/about LinkedIn Profile: http://www.linkedin.com/in/kidehen Personal WebID: http://kingsley.idehen.net/dataspace/person/kidehen#this