Hi all,
the first phase of Wikidata will help to centralize many of the Wikipedia language links. We did a small analysis to figure out the possible impact of this step.
Here are a few highlights: * there are more than 240 million language links in the Wikipedias * they are responsible for about 5 GB of wikitext * on average, 33% of a Wikipedia language edition's wikitext is due to language links - in several Wikipedias, more than 75% of the wikitext are just language links!
The full data set is available here: http://simia.net/languagelinks/
It would be interesting to also figure out how many of the edits are due to changes to the language links. If anyone wants to try that...?
Enjoy the data, Denny
* denny.vrandecic@wikimedia.de wrote:
The full data set is available here: http://simia.net/languagelinks/
http://simia.net/languagelinks/doublelinks/doublelinks.de.html seems to have some errors, for example, it lists "Rundfunkjahr 1924 to [[en:1924 in radio]]" but as far as I can tell, it's there only once (and I don't see it in any template either, nor do there seem to be older revisions with the problem). For http://de.wikipedia.org/wiki/NGC_61 it seems to list many links that are actually commented out in the wikitext. That in fact seems to be a general problem. Is this based on some ad-hoc regex, rather than the database data or a proper parse?
Hello Björn,
yes, you are completely correct, it is based on an ad-hoc reg-exp, and the problem in the examples you mention is indeed due to language links that are commented out. I am surprised by the amount of commented out language links -- there seem to be plenty of them, and I do not fully understand why.
A full parse would have been to expensive to perform. I will update the explanatory text to reflect that. Thank you for finding this issue!
Cheers, Denny
2012/6/25 Bjoern Hoehrmann derhoermi@gmx.net:
- denny.vrandecic@wikimedia.de wrote:
The full data set is available here: http://simia.net/languagelinks/
http://simia.net/languagelinks/doublelinks/doublelinks.de.html seems to have some errors, for example, it lists "Rundfunkjahr 1924 to [[en:1924 in radio]]" but as far as I can tell, it's there only once (and I don't see it in any template either, nor do there seem to be older revisions with the problem). For http://de.wikipedia.org/wiki/NGC_61 it seems to list many links that are actually commented out in the wikitext. That in fact seems to be a general problem. Is this based on some ad-hoc regex, rather than the database data or a proper parse? -- Björn Höhrmann · mailto:bjoern@hoehrmann.de · http://bjoern.hoehrmann.de Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de 25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/
Wikidata-l mailing list Wikidata-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-l
On 25.06.2012 16:39, Denny Vrandečić wrote:
A full parse would have been to expensive to perform. I will update the explanatory text to reflect that. Thank you for finding this issue!
A full parse is out of the question, but stripping comments should be simple enough: /<!--.*?-->/s
-- daniel
I'll maybe... I shouldn't... other stuff to do... gnah...
Let's see. I may well do a new run in the next few days... (you do realize that some of them wikis are pretty big, right?) :)
Cheers, Denny
Am 25. Juni 2012 17:22 schrieb Daniel Kinzler daniel.kinzler@wikimedia.de:
On 25.06.2012 16:39, Denny Vrandečić wrote:
A full parse would have been to expensive to perform. I will update the explanatory text to reflect that. Thank you for finding this issue!
A full parse is out of the question, but stripping comments should be simple enough: /<!--.*?-->/s
-- daniel
-- Daniel Kinzler, Softwarearchitekt
Wikimedia Deutschland e.V. | Eisenacher Straße 2 | 10777 Berlin http://wikimedia.de | Tel. (030) 219 158 260
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985.
Wikidata-l mailing list Wikidata-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-l
Hi Denny,
we extract the inter-language links for DBpedia. Only for five or six languages so far, but I could easily run the extractor for all 111 languages with more than 10000 'good articles'. Shouldn't take more than a few hours. I would use dumps from late May / early June.
We perform a full parse, and while the parser is not 100% accurate, it should be much better than a regex.
The results would be in RDF and look like this:
http://dbpedia.org/resource/Autism http://dbpedia.org/ontology/wikiPageInterLanguageLink http://af.dbpedia.org/resource/Outisme . http://dbpedia.org/resource/Autism http://dbpedia.org/ontology/wikiPageInterLanguageLink http://ar.dbpedia.org/resource/توحد . http://dbpedia.org/resource/Autism http://dbpedia.org/ontology/wikiPageInterLanguageLink http://az.dbpedia.org/resource/Autizm . http://dbpedia.org/resource/Autism http://dbpedia.org/ontology/wikiPageInterLanguageLink http://bn.dbpedia.org/resource/আত্মসংবৃতি . http://dbpedia.org/resource/Autism http://dbpedia.org/ontology/wikiPageInterLanguageLink http://be.dbpedia.org/resource/Аўтызм . http://dbpedia.org/resource/Autism http://dbpedia.org/ontology/wikiPageInterLanguageLink http://be-x-old.dbpedia.org/resource/Аўтызм .
Let me know if that would be useful for you.
Cheers, Christopher
On Mon, Jun 25, 2012 at 5:29 PM, Denny Vrandečić denny.vrandecic@wikimedia.de wrote:
I'll maybe... I shouldn't... other stuff to do... gnah...
Let's see. I may well do a new run in the next few days... (you do realize that some of them wikis are pretty big, right?) :)
Cheers, Denny
Am 25. Juni 2012 17:22 schrieb Daniel Kinzler daniel.kinzler@wikimedia.de:
On 25.06.2012 16:39, Denny Vrandečić wrote:
A full parse would have been to expensive to perform. I will update the explanatory text to reflect that. Thank you for finding this issue!
A full parse is out of the question, but stripping comments should be simple enough: /<!--.*?-->/s
-- daniel
-- Daniel Kinzler, Softwarearchitekt
Wikimedia Deutschland e.V. | Eisenacher Straße 2 | 10777 Berlin http://wikimedia.de | Tel. (030) 219 158 260
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985.
Wikidata-l mailing list Wikidata-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-l
-- Project director Wikidata Wikimedia Deutschland e.V. | Obentrautstr. 72 | 10963 Berlin Tel. +49-30-219 158 26-0 | http://wikimedia.de
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985.
Wikidata-l mailing list Wikidata-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-l
Jona,
yes, that would be useful. If it is not much trouble. Then I would extract the double links from your dataset and provide that as a comparision.
Cheers, Denny
2012/6/25 Jona Christopher Sahnwaldt jc@sahnwaldt.de:
Hi Denny,
we extract the inter-language links for DBpedia. Only for five or six languages so far, but I could easily run the extractor for all 111 languages with more than 10000 'good articles'. Shouldn't take more than a few hours. I would use dumps from late May / early June.
We perform a full parse, and while the parser is not 100% accurate, it should be much better than a regex.
The results would be in RDF and look like this:
http://dbpedia.org/resource/Autism http://dbpedia.org/ontology/wikiPageInterLanguageLink http://af.dbpedia.org/resource/Outisme . http://dbpedia.org/resource/Autism http://dbpedia.org/ontology/wikiPageInterLanguageLink http://ar.dbpedia.org/resource/توحد . http://dbpedia.org/resource/Autism http://dbpedia.org/ontology/wikiPageInterLanguageLink http://az.dbpedia.org/resource/Autizm . http://dbpedia.org/resource/Autism http://dbpedia.org/ontology/wikiPageInterLanguageLink http://bn.dbpedia.org/resource/আত্মসংবৃতি . http://dbpedia.org/resource/Autism http://dbpedia.org/ontology/wikiPageInterLanguageLink http://be.dbpedia.org/resource/Аўтызм . http://dbpedia.org/resource/Autism http://dbpedia.org/ontology/wikiPageInterLanguageLink http://be-x-old.dbpedia.org/resource/Аўтызм .
Let me know if that would be useful for you.
Cheers, Christopher
On Mon, Jun 25, 2012 at 5:29 PM, Denny Vrandečić denny.vrandecic@wikimedia.de wrote:
I'll maybe... I shouldn't... other stuff to do... gnah...
Let's see. I may well do a new run in the next few days... (you do realize that some of them wikis are pretty big, right?) :)
Cheers, Denny
Am 25. Juni 2012 17:22 schrieb Daniel Kinzler daniel.kinzler@wikimedia.de:
On 25.06.2012 16:39, Denny Vrandečić wrote:
A full parse would have been to expensive to perform. I will update the explanatory text to reflect that. Thank you for finding this issue!
A full parse is out of the question, but stripping comments should be simple enough: /<!--.*?-->/s
-- daniel
-- Daniel Kinzler, Softwarearchitekt
Wikimedia Deutschland e.V. | Eisenacher Straße 2 | 10777 Berlin http://wikimedia.de | Tel. (030) 219 158 260
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985.
Wikidata-l mailing list Wikidata-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-l
-- Project director Wikidata Wikimedia Deutschland e.V. | Obentrautstr. 72 | 10963 Berlin Tel. +49-30-219 158 26-0 | http://wikimedia.de
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985.
Wikidata-l mailing list Wikidata-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-l
Wikidata-l mailing list Wikidata-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-l