Hi Daniel, Thanks for the quick and helpful reply. I was hoping that the XML dumps could be changed, but I understand now that the JSON dumps are the recommended format. This is useful, but unfortunately it won't suffice. Wikidata also has pages which are wikitext (for example, https://www.wikidata.org/wiki/Wikidata:WikiProject_Names). These wikitext pages are in the XML dumps, but aren't in the stub dumps nor the JSON dumps. I actually do use these Wikidata wikitext entries to try to reproduce Wikidata in its entirety. So for now, it looks like both XML dumps and JSON dumps will be required. At any rate, thanks again for the excellent reply. On Sat, Nov 26, 2016 at 12:25 PM, Daniel Kinzler <daniel.kinzler(a)wikimedia.de> wrote:

Not sure I follow. Even from a Wikibase on MediaWiki perspective, the XML dumps are still incomplete (since they're missing mainsnak.datatype). For example, consider the following: * You download only the XML dump pages-articles.xml dump from https://dumps.wikimedia.org/wikidatawiki/latest/ * You load it into MediaWiki * You then create a module that looks like the Wikidata Module from Russian Wikipedia: https://ru.wikipedia.org/w/index.php?title=Module:Wikidata&action=edit One line of the file specifically checks for datatype: "if datatype and datatype == 'commonsMedia' then". This line always evaluates to false, even though you are looking at an entity (Q38: Italy) and property (P41: flag image) which does have a datatype for "commonsMedia" (since the XML dump does not have "mainsnak.datatype"). local version of Russian Wikipedia and Wikidata, then all Country infoboxes will not show the country's flag (the above line of code will substitute text for the image) The only way around this is to supplement the XML dump with the JSON dump. But then, you'll need to download 2 large dumps and somehow merge them. (I don't know if MediaWiki has a facility to load the JSON dump, much less merge it) Anyway, I understand that there are technical complications with trying to add mainsnak.datatype to the XML dumps. But if this never gets resolved, then the current situation basically offers two unsatisfying options: * Have an XML dump which is 99.9% complete but still missing key info (mainsnak.datatype) * Try to merge the JSON dump into the XML dump (which MediaWiki may not be able to do) Hope this makes sense. Thanks. On Sun, Nov 27, 2016 at 11:49 AM, Daniel Kinzler <daniel.kinzler(a)wikimedia.de> wrote: > Am 27.11.2016 um 01:15 schrieb gnosygnu: >> This is useful, but unfortunately it won't suffice. Wikidata also has >> pages which are wikitext (for example, >> https://www.wikidata.org/wiki/Wikidata:WikiProject_Names). These >> wikitext pages are in the XML dumps, but aren't in the stub dumps nor >> the JSON dumps. I actually do use these Wikidata wikitext entries to >> try to reproduce Wikidata in its entirety. > > > > -- > Daniel Kinzler > Senior Software Developer > > Wikimedia Deutschland > Gesellschaft zur Förderung Freien Wissens e.V. > > _______________________________________________ > Wikidata mailing list > Wikidata(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikidata

Thanks for all the info. I see my error. I didn't realize that mainsnak.datatype was inferred. I assumed it would have to be embedded directly in the XML's JSON (partly because it is embedded directly in the JSON's dump JSON) The rest of your points make sense. Thanks again for taking the time to clarify. On Mon, Nov 28, 2016 at 11:22 AM, Daniel Kinzler <daniel.kinzler(a)wikimedia.de> wrote:

Honestly, I haven't tried. Just so you know, I'm the developer of XOWA which is an offline wiki app in Java. As such, I'm accessing Wikidata data directly, not through the Wikibase code. (If you're curious, I also use it to recreate Wikidata locally as well. See: http://xowa.org/home/file/screenshot_wikidata.png) Going forward, I'll double-check that my Wikidata issues are not related to my not using Wikibase. Again, my thanks to you for clearing that up. Yup. Wikidata is very cool in concept and in practice. It's amazing to have a single, multi-lingual, verifiable repository of facts / details -- all free and open-content. Kudos to you and your team for the excellent work! On Mon, Nov 28, 2016 at 11:39 AM, Daniel Kinzler <daniel.kinzler(a)wikimedia.de> wrote: > Am 28.11.2016 um 17:34 schrieb gnosygnu: >>> The datatype is implicit, it can be derived from the property ID. You can find >>> it by looking at the Property page's JSON. >>> ... >> >> Thanks for all the info. I see my error. I didn't realize that >> mainsnak.datatype was inferred. I assumed it would have to be embedded >> directly in the XML's JSON (partly because it is embedded directly in >> the JSON's dump JSON) >> >> The rest of your points make sense. Thanks again for taking the time to clarify. > > There may be issues with the import process. > > > > -- > Daniel Kinzler > Senior Software Developer > > Wikimedia Deutschland > Gesellschaft zur Förderung Freien Wissens e.V. > > _______________________________________________ > Wikidata mailing list > Wikidata(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikidata