On 07.09.2015 14:25, Edgard Marx wrote: I am getting an error there. Is the server down maybe? Markus

P.S. If you want to do this yourself to play with it, below is the relevant information on how I wrote this code (looks a bit clumsy in email, but I don't have time now to set up a tutorial page ;-). Markus (1) I modified the example program "EntityStatisticsProcessor" that is part of Wikidata Toolkit [1]. (2) I added a new field to count references: final HashMap<Reference, Integer> refStatistics = new HashMap<>(); (3) The example program already downloads and processes all items and properties in the most recent dump. You just have to add the counting. Essentially, this is the code I run on every ItemDocument and PropertyDocument: public void countReferences(StatementDocument statementDocument) { for (StatementGroup sg : statementDocument.getStatementGroups()) { for (Statement s : sg.getStatements()) { for (Reference r : s.getReferences()) { if (!refStatistics.containsKey(r)) { refStatistics.put(r, 1); } else { refStatistics.put(r, refStatistics.get(r) + 1); } } } } } (the example already has a method "countStatements" that does these iterations, so you can also insert the code there). (4) To print the output to a file, I sort the hash map by values first. Here's some standard code for how to do this: try (PrintStream out = new PrintStream( ExampleHelpers.openExampleFileOuputStream("reference-counts.txt"))) { List<Entry<Reference, Integer>> list = new LinkedList<Entry<Reference, Integer>>( refStatistics.entrySet()); Collections.sort(list, new Comparator<Entry<Reference, Integer>>() { @Override public int compare(Entry<Reference, Integer> o1, Entry<Reference, Integer> o2) { return o2.getValue().compareTo(o1.getValue()); } } ); int singleRefs = 0; for (Entry<Reference, Integer> entry : list) { if (entry.getValue() > 1) { out.println(entry.getValue() + " x " + entry.getKey()); } else { singleRefs++; } } out.println("... and another " + singleRefs + " references that occurred just once."); } catch (IOException e) { e.printStackTrace(); } This code I put into the existing method writeFinalResults() that is called at the end. As I said, this runs in about 30min on my laptop, but downloading the dump file first time takes a bit longer. [1] https://github.com/Wikidata/Wikidata-Toolkit/blob/v0.5.0/wdtk-examples/src/… On 07.09.2015 15:49, Markus Krötzsch wrote:

On 07.09.2015 21:45, Stas Malyshev wrote: Yes, that's the best technique if you already know which reference you are looking for. And it also supports more general patterns, like the Le Figaro one, which is also very interesting. A small fix though: I think you should better use count(?statement) rather than count(?ref), right? I have tried a similar query on the public test endpoint on labs earlier, but it timed out for me (I was using a very common reference though ;-). For rarer references, live queries are definitely the better approach. Markus

Hi, Many thanks for the many answers. I went for the SPARQL solution in the end since I only have a short list of Q-numbers which can be sources. With the new query service it also has the benefit of being something I can just turn into a url and then give to the GLAM to look up on the fly =) Now I also have an excuse to learn some more SPARQL to play around with this =) Cheers, André André Costa | GLAM-tekniker, Wikimedia Sverige | Andre.Costa(a)wikimedia.se | +46 (0)733-964574 Stöd fri kunskap, bli medlem i Wikimedia Sverige. Läs mer på blimedlem.wikimedia.se On 7 September 2015 at 22:29, Stas Malyshev &lt;smalyshev(a)wikimedia.org&gt; wrote: