Update: the first bugs in the export have already been discovered -- and fixed in the script on github. The files I uploaded will be updated on Monday when I have a better upload again (the links file should be fine, the statements file requires a rather tolerant Turtle string literal parser, and the labels file has a malformed line that will hardly work anywhere). Markus On 03/08/13 14:48, Markus Krötzsch wrote: > Hi, > > I am happy to report that an initial, yet fully functional RDF export > for Wikidata is now available. The exports can be created using the > wda-export-data.py script of the wda toolkit [1]. This script > downloads > recent Wikidata database dumps and processes them to create > RDF/Turtle > files. Various options are available to customize the output > (e.g., to > export statements but not references, or to export only texts in > English > and Wolof). The file creation takes a few (about three) hours on my > machine depending on what exactly is exported. > > For your convenience, I have created some example exports based on > yesterday's dumps. These can be found at [2]. There are three Turtle > files: site links only, labels/descriptions/aliases only, statements > only. The fourth file is a preliminary version of the Wikibase > ontology > that is used in the exports. > > The export format is based on our earlier proposal [3], but it adds a > lot of details that had not been specified there yet (namespaces, > references, ID generation, compound datavalue encoding, etc.). > Details > might still change, of course. We might provide regular dumps at > another > location once the format is stable. > > As a side effect of these activities, the wda toolkit [1] is also > getting more convenient to use. Creating code for exporting the data > into other formats is quite easy. > > Features and known limitations of the wda RDF export: > > (1) All current Wikidata datatypes are supported. Commons-media > data is > correctly exported as URLs (not as strings). > > (2) One-pass processing. Dumps are processed only once, even though > this > means that we may not know the types of all properties when we first > need them: the script queries wikidata.org to find missing > information. > This is only relevant when exporting statements. > > (3) Limited language support. The script uses Wikidata's internal > language codes for string literals in RDF. In some cases, this might > not > be correct. It would be great if somebody could create a mapping from > Wikidata language codes to BCP47 language codes (let me know if you > think you can do this, and I'll tell you where to put it) > > (4) Limited site language support. To specify the language of linked > wiki sites, the script extracts a language code from the URL of the > site. Again, this might not be correct in all cases, and it would be > great if somebody had a proper mapping from Wikipedias/Wikivoyages to > language codes. > > (5) Some data excluded. Data that cannot currently be edited is not > exported, even if it is found in the dumps. Examples include > statement > ranks and timezones for time datavalues. I also currently exclude > labels > and descriptions for simple English, formal German, and informal > Dutch, > since these would pollute the label space for English, German, and > Dutch > without adding much benefit (other than possibly for simple English > descriptions, I cannot see any case where these languages should ever > have different Wikidata texts at all). > > Feedback is welcome. > > Cheers, > > Markus > > [1] https://github.com/mkroetzsch/wda > Run "python wda-export.data.py --help" for usage instructions > [2] http://semanticweb.org/RDF/Wikidata/ > [3] http://meta.wikimedia.org/wiki/Wikidata/Development/RDF > _______________________________________________ Wikidata-l mailing list Wikidata-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-l

Hi Sebastian, On 09/08/13 15:44, Sebastian Hellmann wrote: > Hi Markus, > we just had a look at your python code and created a dump. We are > still > getting a syntax error for the turtle dump. You mean "just" as in "at around 15:30 today" ;-)? The code is under heavy development, so changes are quite frequent. Please expect things to be broken in some cases (this is just a little community project, not part of the official Wikidata development). I have just uploaded a new statements export (20130808) to http://semanticweb.org/RDF/Wikidata/ which you might want to try. > > I saw, that you did not use a mature framework for serializing the > turtle. Let me explain the problem: > > Over the last 4 years, I have seen about two dozen people > (undergraduate > and PhD students, as well as Post-Docs) implement "simple" > serializers > for RDF. > > They all failed. > > This was normally not due to the lack of skill, but due to the > lack of > missing time. They wanted to do it quick, but they didn't have the > time > to implement it correctly in the long run. > There are some really nasty problems ahead like encoding or special > characters in URIs. I would direly advise you to: > > 1. use a Python RDF framework > 2. do some syntax tests on the output, e.g. with "rapper" > 3. use a line by line format, e.g. use turtle without prefixes and > just > one triple per line (It's like NTriples, but with Unicode) Yes, URI encoding could be difficult if we were doing it manually. Note, however, that we are already using a standard library for URI encoding in all non-trivial cases, so this does not seem to be a very likely cause of the problem (though some non-zero probability remains). In general, it is not unlikely that there are bugs in the RDF somewhere; please consider this export as an early prototype that is meant for experimentation purposes. If you want an official RDF dump, you will have to wait for the Wikidata project team to get around doing it (this will surely be based on an RDF library). Personally, I already found the dump useful (I successfully imported some 109 million triples of some custom script into an RDF store), but I know that it can require some tweaking. > > We are having a problem currently, because we tried to convert the > dump > to NTriples (which would be handled by a framework as well) with > rapper. > We assume that the error is an extra "<" somewhere (not confirmed) > and > we are still searching for it since the dump is so big.... Ok, looking forward to hear about the results of your search. A good tip for checking such things is to use grep. I did a quick grep on my current local statements export to count the numbers of < and > (this takes less than a minute on my laptop, including on-the-fly decompression). Both numbers were equal, making it unlikely that there is any unmatched < in the current dumps. Then I used grep to check that < and > only occur in the statements files in lines with "commons" URLs. These are created using urllib, so there should never be any < or > in them. > so we can not provide a detailed bug report. If we had one triple per > line, this would also be easier, plus there are advantages for stream > reading. bzip2 compression is very good as well, no need for prefix > optimization. Not sure what you mean here. Turtle prefixes in general seem to be a Good Thing, not just for reducing the file size. The code has no easy way to get rid of prefixes, but if you want a line-by-line export you could subclass my exporter and overwrite the methods for incremental triple writing so that they remember the last subject (or property) and create full triples instead. This would give you a line-by-line export in (almost) no time (some uses of [...] blocks in object positions would remain, but maybe you could live with that). Best wishes, Markus > > All the best, > Sebastian > > Am 03.08.2013 23:22, schrieb Markus Krötzsch: >> Update: the first bugs in the export have already been discovered -- >> and fixed in the script on github. The files I uploaded will be >> updated on Monday when I have a better upload again (the links file >> should be fine, the statements file requires a rather tolerant >> Turtle >> string literal parser, and the labels file has a malformed line that >> will hardly work anywhere). >> >> Markus >> >> On 03/08/13 14:48, Markus Krötzsch wrote: >>> Hi, >>> >>> I am happy to report that an initial, yet fully functional RDF >>> export >>> for Wikidata is now available. The exports can be created using the >>> wda-export-data.py script of the wda toolkit [1]. This script >>> downloads >>> recent Wikidata database dumps and processes them to create >>> RDF/Turtle >>> files. Various options are available to customize the output >>> (e.g., to >>> export statements but not references, or to export only texts in >>> English >>> and Wolof). The file creation takes a few (about three) hours on my >>> machine depending on what exactly is exported. >>> >>> For your convenience, I have created some example exports based on >>> yesterday's dumps. These can be found at [2]. There are three >>> Turtle >>> files: site links only, labels/descriptions/aliases only, >>> statements >>> only. The fourth file is a preliminary version of the Wikibase >>> ontology >>> that is used in the exports. >>> >>> The export format is based on our earlier proposal [3], but it >>> adds a >>> lot of details that had not been specified there yet (namespaces, >>> references, ID generation, compound datavalue encoding, etc.). >>> Details >>> might still change, of course. We might provide regular dumps at >>> another >>> location once the format is stable. >>> >>> As a side effect of these activities, the wda toolkit [1] is also >>> getting more convenient to use. Creating code for exporting the >>> data >>> into other formats is quite easy. >>> >>> Features and known limitations of the wda RDF export: >>> >>> (1) All current Wikidata datatypes are supported. Commons-media >>> data is >>> correctly exported as URLs (not as strings). >>> >>> (2) One-pass processing. Dumps are processed only once, even though >>> this >>> means that we may not know the types of all properties when we >>> first >>> need them: the script queries wikidata.org to find missing >>> information. >>> This is only relevant when exporting statements. >>> >>> (3) Limited language support. The script uses Wikidata's internal >>> language codes for string literals in RDF. In some cases, this >>> might >>> not >>> be correct. It would be great if somebody could create a mapping >>> from >>> Wikidata language codes to BCP47 language codes (let me know if you >>> think you can do this, and I'll tell you where to put it) >>> >>> (4) Limited site language support. To specify the language of >>> linked >>> wiki sites, the script extracts a language code from the URL of the >>> site. Again, this might not be correct in all cases, and it >>> would be >>> great if somebody had a proper mapping from >>> Wikipedias/Wikivoyages to >>> language codes. >>> >>> (5) Some data excluded. Data that cannot currently be edited is not >>> exported, even if it is found in the dumps. Examples include >>> statement >>> ranks and timezones for time datavalues. I also currently exclude >>> labels >>> and descriptions for simple English, formal German, and informal >>> Dutch, >>> since these would pollute the label space for English, German, and >>> Dutch >>> without adding much benefit (other than possibly for simple English >>> descriptions, I cannot see any case where these languages should >>> ever >>> have different Wikidata texts at all). >>> >>> Feedback is welcome. >>> >>> Cheers, >>> >>> Markus >>> >>> [1] https://github.com/mkroetzsch/wda >>> Run "python wda-export.data.py --help" for usage instructions >>> [2] http://semanticweb.org/RDF/Wikidata/ >>> [3] http://meta.wikimedia.org/wiki/Wikidata/Development/RDF >>> >> >> >> _______________________________________________ >> Wikidata-l mailing list >> Wikidata-l(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/wikidata-l >> > >

Apart from the above, doesn't wgLanguageCode in https://noc.wikimedia.org/conf/highlight.php?file=InitialiseSettings.php have what you need?

* Wikidata uses "be-x-old" as a code, but MediaWiki messages for this language seem to use "be-tarask" as a language code. So there must be a mapping somewhere. Where?

the language list (variable siteLanguageCodes).

> mapping somewhere. Where? Where I linked it.

> provides some mappings. For example, it maps "zh-yue" to "yue". Yet, > Wikidata use both of these codes. What does this mean? > > Answers to Nemo's points inline: > > On 04/08/13 06:15, Federico Leva (Nemo) wrote: >> Markus Krötzsch, 03/08/2013 15:48:

are considering wrong.

Hi Markus, Our code 'sr-ec' is at this moment effectively equivalent to 'sr-Cyrl', likewise is our code 'sr-el' currently effectively equivalent to 'sr-Latn'. Both might change, once dialect codes of Serbian are added to the IANA subtag registry at http://www.iana.org/assignments/language-subtag-registry/language-subtag-re… Our code 'nrm' is not being used for the Narom language as ISO 639-3 does, see: http://www-01.sil.org/iso639-3/documentation.asp?id=nrm We rather use it for the Norman / Nourmaud, as described in http://en.wikipedia.org/wiki/Norman_language The Norman language is recognized by the linguist list and many others but as of yet not present in ISO 639-3. It should probably be suggested to be added. We should probaly map it to a private code meanwhile. Our code 'ksh' is currently being used to represent a superset of what it stands for in ISO 639-3. Since ISO 639 lacks a group code for Ripuarian, we use the code of the only Ripuarian variety (of dozens) having a code, to represent the whole lot. We should probably suggest to add a group code to ISO 639, and at least the dozen+ Ripuarian languages that we are using, and map 'ksh' to a private code for Ripuarian meanwhile. Note also, that for the ALS/GSW and the KSH Wikipedias, page titles are not guaranteed to be in the languages of the Wikipedias. They are often in German instead. Details to be found in their respective page titleing rules. Moreover, for the ksh Wikipedia, unlike some other multilingual or multidialectal Wikipedias, texts are not, or quite often incorrectly, labelled as belonging to a certain dialect. See also: http://meta.wikimedia.org/wiki/Special_language_codes Greetings -- Purodha *Gesendet:* Sonntag, 04. August 2013 um 19:01 Uhr *Von:* "Markus Krötzsch" &lt;markus(a)semantic-mediawiki.org&gt; *An:* "Federico Leva (Nemo)" &lt;nemowiki(a)gmail.com&gt; *Cc:* "Discussion list for the Wikidata project." &lt;wikidata-l(a)lists.wikimedia.org&gt; *Betreff:* [Wikidata-l] Wikidata language codes (Was: Wikidata RDF export available) Small update: I went through the language list at https://github.com/mkroetzsch/wda/blob/master/includes/epTurtleFileWriter.p… and added a number of TODOs to the most obvious problematic cases. Typical problems are: * Malformed language codes ('tokipona') * Correctly formed language codes without any official meaning (e.g., 'cbk-zam') * Correctly formed codes with the wrong meaning (e.g., 'sr-ec': Serbian from Ecuador?!) * Language codes with redundant information (e.g., 'kk-cyrl' should be the same as 'kk' according to IANA, but we have both) * Use of macrolanguages instead of languages (e.g., "zh" is not "Mandarin" but just "Chinese"; I guess we mean Mandarin; less sure about Kurdish ...) * Language codes with incomplete information (e.g., "sr" should be "sr-Cyrl" or "sr-Latn", both of which already exist; same for "zh" and "zh-Hans"/"zh-Hant", but also for "zh-HK" [is this simplified or traditional?]). I invite any language experts to look at the file and add comments/improvements. Some of the issues should possibly also be considered on the implementation side: we don't want two distinct codes for the same thing. Cheers, Markus On 04/08/13 16:35, Markus Krötzsch wrote: > On 04/08/13 13:17, Federico Leva (Nemo) wrote: >> Markus Krötzsch, 04/08/2013 12:32: >>> * Wikidata uses "be-x-old" as a code, but MediaWiki messages for this >>> language seem to use "be-tarask" as a language code. So there must be a >>> mapping somewhere. Where? >> >> Where I linked it. > > Are you sure? The file you linked has mappings from site ids to language > codes, not from language codes to language codes. Do you mean to say: > "If you take only the entries of the form 'XXXwiki' in the list, and > extract a language code from the XXX, then you get a mapping from > language codes to language codes that covers all exceptions in > Wikidata"? This approach would give us: > > 'als' : 'gsw', > 'bat-smg': 'sgs', > 'be_x_old' : 'be-tarask', > 'crh': 'crh-latn', > 'fiu_vro': 'vro', > 'no' : 'nb', > 'roa-rup': 'rup', > 'zh-classical' : 'lzh' > 'zh-min-nan': 'nan', > 'zh-yue': 'yue' > > Each of the values on the left here also occur as language tags in > Wikidata, so if we map them, we use the same tag for things that > Wikidata has distinct tags for. For example, Q27 has a label for yue but > also for zh-yue [1]. It seems to be wrong to export both of these with > the same language tag if Wikidata uses them for different purposes. > > Maybe this is a bug in Wikidata and we should just not export texts with > any of the above codes at all (since they always are given by another > tag directly)? > >> >>> * MediaWiki's http://www.mediawiki.org/wiki/Manual:$wgDummyLanguageCodes >>> provides some mappings. For example, it maps "zh-yue" to "yue". Yet, >>> Wikidata use both of these codes. What does this mean? >>> >>> Answers to Nemo's points inline: >>> >>> On 04/08/13 06:15, Federico Leva (Nemo) wrote: >>>> Markus Krötzsch, 03/08/2013 15:48: > > ... > >>>> Apart from the above, doesn't wgLanguageCode in >>>> https://noc.wikimedia.org/conf/highlight.php?file=InitialiseSettings.php >>>> >>>> have what you need? >>> >>> Interesting. However, the list there does not contain all 300 sites that >>> we currently find in Wikidata dumps (and some that we do not find there, >>> including things like dkwiki that seem to be outdated). The full list of >>> sites we support is also found in the file I mentioned above, just after >>> the language list (variable siteLanguageCodes). >> >> Of course not all wikis are there, that configuration is needed only >> when the subdomain is "wrong". It's still not clear to me what codes you >> are considering wrong. > > Well, the obvious: if a language used in Wikidata labels or on Wikimedia > sites has an official IANA code [2], then we should use this code. Every > other code would be "wrong". For languages that do not have any accurate > code, we should probably use a private code, following the requirements > of BCP 47 for private use subtags (in particular, they should have a > single x somewhere). > > This does not seem to be done correctly by my current code. For example, > we now map 'map_bmswiki' to 'map-bms'. While both 'map' and 'bms' are > lANA language tags, I am not sure that their combination makes sense. > The language should be Basa Banyumasan, but bms is for Bilma Kanuri (and > it is a language code, not a dialect code). Note that map-bms does not > occur in the file you linked to, so I guess there is some more work to do. > > Markus > > [1] http://www.wikidata.org/wiki/Special:Export/Q27 > [2] > http://www.iana.org/assignments/language-subtag-registry/language-subtag-re… > > > _______________________________________________ Wikidata-l mailing list Wikidata-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-l _______________________________________________ Wikidata-l mailing list Wikidata-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-l