Hi,

as some of you may noticed we recently had huge troubles[0] with Wikidata's Q183 (the item about Germany). The immediate cause of these problems was the huge size of that item.

Below I'm going to explain what led to the problems and how I was able to workaround the problem for now.

As can been seen in the item's edit history[1], the item hadn't been edited since early July, until HHVM had been enabled on Wikidata. I can't tell for sure, but I presume the item had been unchangeable since July, because of it's size. While the huge size of the item presumably prevented editing, it never was a (at least on a huge scale) noticeable problem viewing it and using it in the client.

After HHVM got enabled on Wikidata it was, due to HHVM being way faster than Zend PHP, possible to change the item again. This was done once[1] and led to the item internally being converted into our new serialization format (this can be seen in the huge size change of that small content change). The now much larger item then became a problem for the slower Zend PHP (in both the client and repo) because it was much larger then the old version.

As explained above the item's size led to out of memory errors, php segfaults, exceptions and various other problems causing huge disruption within both Wikidata and the Wikipedias. Because of the huge impact of the problem I was forced to find a way around the immediate problems in a timely manner.

I started mitigating the problems by simply undoing the change to the new serialization format and restored the version from July 8[1]. To my own surprise even that version didn't render, probably because our DataModel is less efficient now then it used to be. After that I pulled out an older (and thus smaller) version[2] from the edit history which I could view (with oldid=120566337). But sadly also this revision, although much smaller than what we used to have, didn't always render (it might have worked sometimes and probably caused a little less trouble in other components). After all I was forced to revert to revision 116786096[3] which has way less data then what used to be on the item. Using that revision the item could be rendered again and can also be used by other components again (eg. in the Wikipedias).

As any change to that item would trigger an update of the serialization format again (which would also make the stored data much bigger), I (together with Marc-André Pelletier (coren)) decided that the item needs to be "freezed" in a working state, so that it can't be altered (that also makes sure we can go back to the version with the most data once this has been solved). Because of that we chose to superprotect the item to make sure it can't be edited by anyone[4].

The above steps work as a temporary workaround to the problems caused in all components, but of course they are only a temporary countermeasure and we will need to fix this properly.

Cheers,

Marius

[0]: https://bugzilla.wikimedia.org/show_bug.cgi?id=71519 and others [1]: https://www.wikidata.org/wiki/Q183?action=history [2]: https://www.wikidata.org/w/index.php?title=Q183&oldid=120566337 [3]: https://www.wikidata.org/w/index.php?title=Q183&oldid=116786096 [4]: https://www.wikidata.org/w/index.php?oldid=162225140#Temporary_protection_of...