It certainly helps, however, I think Aidan's suggestion goes into the direction of having an official dump distribution. Imagine how many CO2 can be spared just by avoiding the computational resource to recreate this dump every time ones need it. Besides, it standardise the dataset used for research purposes. On Wed, Dec 18, 2019, 11:26 Marco Fossati <fossati(a)spaziodati.eu> wrote:

Hi all, Yes, Benno's WDumper could be used for this purpose. The motivation for the whole project was very similar to what Aidan describes. We realised thought that there won't be a single good way to build smaller dump that would serve every conceivable use in research, which is why the UI let's users make custom dumps. In general, we are happy to hear more ideas on how to build useful smaller dumps that would be interesting. We are also accepting pull requests. Benno, could you add a feature to include only items with Wikipedia page (in some language, or in any language)? Edgard, I don't think making this more "official" will be very important for most researchers. Benno spent quite some time on aligning the RDF export with the official dumps, so in practice, WDumper mostly produces a subset of the triples of the official dump (which one could also have extracted manually). If there are differences left between the formats, we will be happy to hear about them (a github issue would be the best way to report it). As Benno already wrote, WDumper connects to Zenodo to ensure that exported datasets are archived in a permanent and citable fashion. This is very important for research. As far as I know, none of the existing dumps (official or not) guarantee long-term availability at the moment. Cheers, Markus On 18/12/2019 13:37, Edgard Marx wrote:

Hey all, Just a general response to all the comments thus far. - @Marco et al., regarding the WDumper by Benno, this is a very cool initiative! In fact I spotted it just *after* posting so I think this goes quite some ways towards addressing the general issue raised. - @Markus, I partially disagree regarding the importance of rubber-stamping a "notable dump" on the Wikidata side. I would see it's value as being something like the "truthy dump", which I believe has been widely used in research for working with a concise sub-set of Wikidata. Perhaps a middle ground is for a sporadic "notable dump" to be generated by WDumper and published on Zenodo. This may be sufficient in terms of making the dump available and reusable for research purposes (or even better than the current dumps, given the permanence you mention). Also it would reduce costs on the Wikidata side (I don't think a notable dump would be necessary to generate on a weekly basis, for example). - @Lydia, good point! I was thinking that filtering by wikilinks will just drop some more obscure nodes (like Q51366847 for example), but had not considered that there are some more general "concepts" that do not have a corresponding Wikipedia article. All the same, in a lot of the research we use Wikidata for, we are not particularly interested in one thing or another, but more interested in facilitating what other people are interested in. Examples would be query performance, finding paths, versioning, finding references, etc. But point taken! Maybe there is a way to identify "general entities" that do not have wikilinks, but do have a high degree or centrality, for example? Would a degree-based or centrality-based filter be possible in something like WDumper (perhaps it goes beyond the original purpose; certainly it does not seem trivial in terms of resources used)? Would it be a good idea? In summary, I like the idea of using WDumper to sporadically generate -- and publish on Zenodo -- a "notable version" of Wikidata filtered by sitelinks (perhaps also allowing other high-degree or high-PageRank nodes to pass the filter). At least I know I would use such a dump. Best, Aidan On 2019-12-19 6:46, Lydia Pintscher wrote:

On Sat, 21 Dec 2019 at 17:25, Lydia Pintscher <Lydia.Pintscher(a)wikimedia.de> wrote: Yeah, there are a bunch of quite varied motivations for subsets. I have found the topic of Wikidata subsetting and data dumps coming up again and again. Most recently in a lifescience/bioinformations setting which is how we ended up collecting raw materials in the doc already shared here, https://docs.google.com/document/d/1MmrpEQ9O7xA6frNk6gceu_IbQrUiEYGI9vcQjDv… but also in other domains. If people here care to drop use cases, thoughts and notes (*however scrappy*) into that doc I will make a pass over it to try to pull together a more readable summary of the various motivations for subsetting. The work Adam wrote up at https://addshore.com/2019/10/your-own-wikidata-query-service-with-no-limits… is also very relevant... (I think this is something That is also a fine place to record things! I don’t mean to fork the discussion. Maybe we could have a call for interested parties in the new year? Dan

Hi Aidan, since DBpedia has been around for twelve years now, we spent the last 3 years intensively re-engineering to solve problems like this. Last week, we finished the Virtuoso DBpedia Docker[1]  to work on Databus Collections[2],[3]. Databus contains different repartitions of all the datasets, i.e. Wikipedia/Wikidata extractions and external data. The idea here is that datasets or graphs are stored in a granular manner and then you make your own collection (DCAT Catalog) or re-use collections by others. This will go in the direction to build 1 Billion derived Knowledge Graphs until 2025: https://databus.dbpedia.org/dbpedia/publication/strategy/2019.09.09/strateg… We analysed a lot of problems in the GlobalFactSync project [5] and studied Wikidata intensively. Our conclusion here is that we will invert DBpedia by 180 degrees in the future. So instead of taking the main data from Wikipedia and Wikidata, we will take it from the sources directly, since all data in Wikipedia and Wikidata comes from somewhere else. So the new direction is LOD -> DBpedia -> Wikipedia/Wikidata via sameAs,equivalentClass/Property mappings. This is not a solution for the dump size problem per se, because we are creating even bigger and more varied and domain-specific knowledge graphs and dumps via FlexiFusion. Besides the flexible source partitions, we offer a partition by property, where you can simply pick the properties you want for your knowledge graph and then docker-load it into the SPARQL store of choice. There is no manual yet, but this query gives you all 3.8 million birthdates of the new big fused graph: https://databus.dbpedia.org/yasgui/ PREFIX dataid: <http://dataid.dbpedia.org/ns/core#> PREFIX dataid-cv: <http://dataid.dbpedia.org/ns/cv#> PREFIX dct: <http://purl.org/dc/terms/> PREFIX dcat:  <http://www.w3.org/ns/dcat#> SELECT DISTINCT ?file WHERE {      ?dataset dataid:version <https://databus.dbpedia.org/vehnem/flexifusion/fusion/2019.11.15> .     ?dataset dcat:distribution ?distribution .     ?distribution <http://dataid.dbpedia.org/ns/cv#tag> 'birthDate'^^<http://www.w3.org/2001/XMLSchema#string> .     ?distribution dcat:downloadURL ?file . } Where you can filter more here: https://databus.dbpedia.org/vehnem/flexifusion/fusion/2019.11.15 During the next year, we will include all European library data into the syncing process, several national statistical datasets and other data and refine the way to extract exactly the partition you need. It is an opportunistic extension to Linked Open Data, where you can select the partition you need independent of the IDs or vocab used. -- Sebastian [1] https://github.com/dbpedia/Dockerized-DBpedia [2] https://forum.dbpedia.org/t/dbpedia-dataset-2019-08-30-pre-release/219 [3] https://github.com/dbpedia/minimal-download-client [4] https://svn.aksw.org/papers/2019/ISWC_FlexiFusion/public.pdf [5] https://meta.wikimedia.org/wiki/Grants:Project/DBpedia/GlobalFactSyncRE On 19.12.19 23:15, Aidan Hogan wrote: -- All the best, Sebastian Hellmann Director of Knowledge Integration and Linked Data Technologies (KILT) Competence Center at the Institute for Applied Informatics (InfAI) at Leipzig University Executive Director of the DBpedia Association Projects: http://dbpedia.org, http://nlp2rdf.org, http://linguistics.okfn.org, https://www.w3.org/community/ld4lt <http://www.w3.org/community/ld4lt> Homepage: http://aksw.org/SebastianHellmann Research Group: http://aksw.org