Good morning. I just found a bug that was caused
by a bug in the
Wikidata dumps (a value that should be a URI was not). This led to a
few dozen lines with illegal qnames of the form "w: ". The updated
script fixes this.
Cheers,
Markus
On 09/08/13 18:15, Markus Krötzsch wrote:
Hi Sebastian,
On 09/08/13 15:44, Sebastian Hellmann wrote:
Hi Markus,
we just had a look at your python code and created a dump. We are still
getting a syntax error for the turtle dump.
You mean "just" as in "at around 15:30 today" ;-)? The code is under
heavy development, so changes are quite frequent. Please expect things
to be broken in some cases (this is just a little community project, not
part of the official Wikidata development).
I have just uploaded a new statements export (20130808) to
http://semanticweb.org/RDF/Wikidata/ which you might want to try.
I saw, that you did not use a mature framework for serializing the
turtle. Let me explain the problem:
Over the last 4 years, I have seen about two dozen people
(undergraduate
and PhD students, as well as Post-Docs) implement "simple" serializers
for RDF.
They all failed.
This was normally not due to the lack of skill, but due to the lack of
missing time. They wanted to do it quick, but they didn't have the time
to implement it correctly in the long run.
There are some really nasty problems ahead like encoding or special
characters in URIs. I would direly advise you to:
1. use a Python RDF framework
2. do some syntax tests on the output, e.g. with "rapper"
3. use a line by line format, e.g. use turtle without prefixes and just
one triple per line (It's like NTriples, but with Unicode)
Yes, URI encoding could be difficult if we were doing it manually. Note,
however, that we are already using a standard library for URI encoding
in all non-trivial cases, so this does not seem to be a very likely
cause of the problem (though some non-zero probability remains). In
general, it is not unlikely that there are bugs in the RDF somewhere;
please consider this export as an early prototype that is meant for
experimentation purposes. If you want an official RDF dump, you will
have to wait for the Wikidata project team to get around doing it (this
will surely be based on an RDF library). Personally, I already found the
dump useful (I successfully imported some 109 million triples of some
custom script into an RDF store), but I know that it can require some
tweaking.
We are having a problem currently, because we tried to convert the dump
to NTriples (which would be handled by a framework as well) with
rapper.
We assume that the error is an extra "<" somewhere (not confirmed) and
we are still searching for it since the dump is so big....
Ok, looking forward to hear about the results of your search. A good tip
for checking such things is to use grep. I did a quick grep on my
current local statements export to count the numbers of < and > (this
takes less than a minute on my laptop, including on-the-fly
decompression). Both numbers were equal, making it unlikely that there
is any unmatched < in the current dumps. Then I used grep to check that
< and > only occur in the statements files in lines with "commons" URLs.
These are created using urllib, so there should never be any < or > in
them.
so we can not provide a detailed bug report. If
we had one triple per
line, this would also be easier, plus there are advantages for stream
reading. bzip2 compression is very good as well, no need for prefix
optimization.
Not sure what you mean here. Turtle prefixes in general seem to be a
Good Thing, not just for reducing the file size. The code has no easy
way to get rid of prefixes, but if you want a line-by-line export you
could subclass my exporter and overwrite the methods for incremental
triple writing so that they remember the last subject (or property) and
create full triples instead. This would give you a line-by-line export
in (almost) no time (some uses of [...] blocks in object positions would
remain, but maybe you could live with that).
Best wishes,
Markus
All the best,
Sebastian
Am 03.08.2013 23:22, schrieb Markus Krötzsch:
> Update: the first bugs in the export have already been discovered --
> and fixed in the script on github. The files I uploaded will be
> updated on Monday when I have a better upload again (the links file
> should be fine, the statements file requires a rather tolerant Turtle
> string literal parser, and the labels file has a malformed line that
> will hardly work anywhere).
>
> Markus
>
> On 03/08/13 14:48, Markus Krötzsch wrote:
>> Hi,
>>
>> I am happy to report that an initial, yet fully functional RDF export
>> for Wikidata is now available. The exports can be created using the
>> wda-export-data.py script of the wda toolkit [1]. This script
>> downloads
>> recent Wikidata database dumps and processes them to create
>> RDF/Turtle
>> files. Various options are available to customize the output
>> (e.g., to
>> export statements but not references, or to export only texts in
>> English
>> and Wolof). The file creation takes a few (about three) hours on my
>> machine depending on what exactly is exported.
>>
>> For your convenience, I have created some example exports based on
>> yesterday's dumps. These can be found at [2]. There are three Turtle
>> files: site links only, labels/descriptions/aliases only, statements
>> only. The fourth file is a preliminary version of the Wikibase
>> ontology
>> that is used in the exports.
>>
>> The export format is based on our earlier proposal [3], but it adds a
>> lot of details that had not been specified there yet (namespaces,
>> references, ID generation, compound datavalue encoding, etc.).
>> Details
>> might still change, of course. We might provide regular dumps at
>> another
>> location once the format is stable.
>>
>> As a side effect of these activities, the wda toolkit [1] is also
>> getting more convenient to use. Creating code for exporting the data
>> into other formats is quite easy.
>>
>> Features and known limitations of the wda RDF export:
>>
>> (1) All current Wikidata datatypes are supported. Commons-media
>> data is
>> correctly exported as URLs (not as strings).
>>
>> (2) One-pass processing. Dumps are processed only once, even though
>> this
>> means that we may not know the types of all properties when we first
>> need them: the script queries
wikidata.org to find missing
>> information.
>> This is only relevant when exporting statements.
>>
>> (3) Limited language support. The script uses Wikidata's internal
>> language codes for string literals in RDF. In some cases, this might
>> not
>> be correct. It would be great if somebody could create a mapping from
>> Wikidata language codes to BCP47 language codes (let me know if you
>> think you can do this, and I'll tell you where to put it)
>>
>> (4) Limited site language support. To specify the language of linked
>> wiki sites, the script extracts a language code from the URL of the
>> site. Again, this might not be correct in all cases, and it would be
>> great if somebody had a proper mapping from Wikipedias/Wikivoyages to
>> language codes.
>>
>> (5) Some data excluded. Data that cannot currently be edited is not
>> exported, even if it is found in the dumps. Examples include
>> statement
>> ranks and timezones for time datavalues. I also currently exclude
>> labels
>> and descriptions for simple English, formal German, and informal
>> Dutch,
>> since these would pollute the label space for English, German, and
>> Dutch
>> without adding much benefit (other than possibly for simple English
>> descriptions, I cannot see any case where these languages should ever
>> have different Wikidata texts at all).
>>
>> Feedback is welcome.
>>
>> Cheers,
>>
>> Markus
>>
>> [1]
https://github.com/mkroetzsch/wda
>> Run "python wda-export.data.py --help" for usage instructions
>> [2]
http://semanticweb.org/RDF/Wikidata/
>> [3]
http://meta.wikimedia.org/wiki/Wikidata/Development/RDF
>>
>
>
> _______________________________________________
> Wikidata-l mailing list
> Wikidata-l(a)lists.wikimedia.org
>
https://lists.wikimedia.org/mailman/listinfo/wikidata-l
>