New subject: Critical JSON bugs back in production; all dumps broken now

4 Aug 2015

Hi,

The recent Wikidata JSON dumps again contain huge amounts of broken JSON 
where empty maps are serialized as [] instead of using {}. Just grep for

"claims":[]
or
"aliases":[]
or
any other key that requires a map

to find many examples. The scope of the problem is massive. Basically 
all entity documents that include some empty map are broken, which is 
almost every entity document in 
http://dumps.wikimedia.org/other/wikidata/20150803.json.gz. Concretely, 
there are around 15.7 million entities with [] for aliases.

This is critically breaking the consumption of Wikidata content for all 
model-based JSON parsers, including Wikidata Toolkit.

The bug used to occur only in XML dumps, but now also affects the JSON 
dumps in the same way. In previous JSON dumps, the problem was avoided 
by omitting empyt maps altogether (no keys, no values), which is better 
because it allows implementations to fall back to the obvious default. 
This is still done in the Web API, e.g., 
https://www.wikidata.org/wiki/Special:EntityData/Q12062430.json

It would be nice to test the export code before deploying it.

Regards,

Markus