Hey folks :)
Are you a tool author relying on dumps or Special:Export or use Lua on Wikidata to access data in items? Please read on. The JSON format returned by the API and what we use to store the data internally has been different for a long time. This has been a headache as people had to deal with two different JSON serializations. We've fixed this now to only use one. This means the format you get in the XML dumps, via Special:Export and when accessing the raw item page content via Lua will change to be the same as what you get from the API. We plan to deploy this change on August 26th.
This is one of the remaining blockers for statements on properties and further progress on Commons support.
I'm sorry for the disruption. If you need help with adapting your tools please let me know.
Cheers Lydia
How about the XML format returned by "api.php?action=query&prop=revisions&rvprop=content&format=xml"? What will it change to?
For example: The current return value of http://www.wikidata.org/w/api.php?action=query&format=xml&titles=Q1&... is: <api> <query> <pages> <page pageid="129" ns="0" title="Q1"> <revisions> <rev contentformat="application/json" contentmodel="wikibase-item" xml:space="preserve"> {"label":{"en":"universe","fr":"Univers", ......
Is it going to become: <api> <query> <pages> <page pageid="129" ns="0" title="Q1"> <revisions> <rev contentformat="application/json" contentmodel="wikibase-item" xml:space="preserve">
{"labels":{"en":{"language":"en","value":"universe"},"fr":{"language":"fr","value":"Univers"}, ......
Is there a way that I can try this BREAKING CHANGE before Aug 26? For example, can I see the changed JSON format at test.wikidata.org before Aug 26?
Thanks, Huidong
On Fri, Aug 15, 2014 at 11:04 AM, Lydia Pintscher < lydia.pintscher@wikimedia.de> wrote:
Hey folks :)
Are you a tool author relying on dumps or Special:Export or use Lua on Wikidata to access data in items? Please read on. The JSON format returned by the API and what we use to store the data internally has been different for a long time. This has been a headache as people had to deal with two different JSON serializations. We've fixed this now to only use one. This means the format you get in the XML dumps, via Special:Export and when accessing the raw item page content via Lua will change to be the same as what you get from the API. We plan to deploy this change on August 26th.
This is one of the remaining blockers for statements on properties and further progress on Commons support.
I'm sorry for the disruption. If you need help with adapting your tools please let me know.
Cheers Lydia
-- Lydia Pintscher - http://about.me/lydia.pintscher Product Manager for Wikidata
Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985.
Wikidata-l mailing list Wikidata-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-l
On Tue, Aug 19, 2014 at 7:29 AM, Huidong Zhang anthonyzhang@google.com wrote:
How about the XML format returned by "api.php?action=query&prop=revisions&rvprop=content&format=xml"? What will it change to?
No those will not change.
For example: The current return value of http://www.wikidata.org/w/api.php?action=query&format=xml&titles=Q1&... is:
<api> <query> <pages> <page pageid="129" ns="0" title="Q1"> <revisions> <rev contentformat="application/json" contentmodel="wikibase-item" xml:space="preserve"> {"label":{"en":"universe","fr":"Univers", ......
Is it going to become:
<api> <query> <pages> <page pageid="129" ns="0" title="Q1"> <revisions> <rev contentformat="application/json" contentmodel="wikibase-item" xml:space="preserve">
{"labels":{"en":{"language":"en","value":"universe"},"fr":{"language":"fr","value":"Univers"}, ......
Is there a way that I can try this BREAKING CHANGE before Aug 26? For example, can I see the changed JSON format at test.wikidata.org before Aug 26?
Yes it should be there from Thursday night on.
Cheers Lydia
Another related question: what are the new content model strings used for the new format?
Cheers,
Markus
On 15.08.2014 20:04, Lydia Pintscher wrote:
Hey folks :)
Are you a tool author relying on dumps or Special:Export or use Lua on Wikidata to access data in items? Please read on. The JSON format returned by the API and what we use to store the data internally has been different for a long time. This has been a headache as people had to deal with two different JSON serializations. We've fixed this now to only use one. This means the format you get in the XML dumps, via Special:Export and when accessing the raw item page content via Lua will change to be the same as what you get from the API. We plan to deploy this change on August 26th.
This is one of the remaining blockers for statements on properties and further progress on Commons support.
I'm sorry for the disruption. If you need help with adapting your tools please let me know.
Cheers Lydia
Hi Lydia,
Thanks for your quick reply.
So the data stored at wikidata.org repository database will not change, but the dumps of the database will change? Is it possible to provide 2 kinds of dumps? One for raw data (in internal JSON serialization format) and one for data model (in exteranl JSON serialization format).
Thanks, Huidong
On Wed, Aug 20, 2014 at 4:54 AM, Markus Krötzsch < markus@semantic-mediawiki.org> wrote:
Another related question: what are the new content model strings used for the new format?
Cheers,
Markus
On 15.08.2014 20:04, Lydia Pintscher wrote:
Hey folks :)
Are you a tool author relying on dumps or Special:Export or use Lua on Wikidata to access data in items? Please read on. The JSON format returned by the API and what we use to store the data internally has been different for a long time. This has been a headache as people had to deal with two different JSON serializations. We've fixed this now to only use one. This means the format you get in the XML dumps, via Special:Export and when accessing the raw item page content via Lua will change to be the same as what you get from the API. We plan to deploy this change on August 26th.
This is one of the remaining blockers for statements on properties and further progress on Commons support.
I'm sorry for the disruption. If you need help with adapting your tools please let me know.
Cheers Lydia
Wikidata-l mailing list Wikidata-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-l
On Wed, Aug 20, 2014 at 6:27 PM, Huidong Zhang anthonyzhang@google.com wrote:
Hi Lydia,
Thanks for your quick reply.
So the data stored at wikidata.org repository database will not change, but the dumps of the database will change?
The internal format will change to be the same as the external one. So what is stored in the database will change. This is also what you get through the XML dumps (not the JSON dumps). So those will also change. What will not change is the external format. This is what you get through the API for example.
Is it possible to provide 2 kinds of dumps? One for raw data (in internal JSON serialization format) and one for data model (in exteranl JSON serialization format).
The whole point of the exercise is to make them the same ;-) The JSON dumps at http://dumps.wikimedia.org/other/wikidata/ are what we recommend using and they will not change. Let me know if there is anything that's still unclear. I know it's a bit confusing and we should have gone with one serialization format from the start.
Cheers Lydia
Hey folks :)
On Fri, Aug 15, 2014 at 8:04 PM, Lydia Pintscher lydia.pintscher@wikimedia.de wrote:
Hey folks :)
Are you a tool author relying on dumps or Special:Export or use Lua on Wikidata to access data in items? Please read on. The JSON format returned by the API and what we use to store the data internally has been different for a long time. This has been a headache as people had to deal with two different JSON serializations. We've fixed this now to only use one. This means the format you get in the XML dumps, via Special:Export and when accessing the raw item page content via Lua will change to be the same as what you get from the API. We plan to deploy this change on August 26th.
This is one of the remaining blockers for statements on properties and further progress on Commons support.
I'm sorry for the disruption. If you need help with adapting your tools please let me know.
This change is now live on test.wikidata.org. Please take some time this weekend and help test this thoroughly. We've done our best but something might have slipped through and it'd be pretty bad if we don't catch it before it goes on wikidata.org. Edit a bit and let me know about anything strange please.
Cheers Lydia