Gift season! We have launched structured data on Commons, available from all wikis.
TLDR; One data store. Use everywhere. Upload table data to Commons, with localization, and use it to create wiki tables, lists, or use directly in graphs. Works for GeoJSON maps too. Must be licensed as CC0. Try this per-state GDP map demo, and select multiple years. More demos at the bottom. US Map state highlight https://en.wikipedia.org/wiki/Template:Graph:US_Map_state_highlight
Data can now be stored as *.tab and *.map pages in the data namespace on Commons. That data may contain localization, so a table cell could be in multiple languages. And that data is accessible from any wikis, by Lua scripts, Graphs, and Maps.
Lua lets you generate wiki tables from the data by filtering, converting, mixing, and formatting the raw data. Lua also lets you generate lists. Or any wiki markup.
Graphs can use both .tab and .map directly to visualize the data and let users interact with it. The GDP demo above uses a map from Commons, and colors each segment with the data based on a data table.
Kartographer (<maplink>/<mapframe>) can use the .map data as an extra layer on top of the base map. This way we can show endangered species' habitat.
== Demo == * Raw data example https://commons.wikimedia.org/wiki/Data:Weather/New_York_City.tab * Interactive Weather data https://en.wikipedia.org/wiki/Template:Graph:Weather_monthly_history * Same data in Weather template https://en.wikipedia.org/wiki/User:Yurik/WeatherDemo * Interactive GDP map https://en.wikipedia.org/wiki/Template:Graph:US_Map_state_highlight * Endangered Jemez Mountains salamander - habitat https://en.wikipedia.org/wiki/Jemez_Mountains_salamander#/maplink/0 * Population history https://en.wikipedia.org/wiki/Template:Graph:Population_history * Line chart https://en.wikipedia.org/wiki/Template:Graph:Lines
== Getting started == * Try creating a page at data:Sandbox/<user>.tab on Commons. Don't forget the .tab extension, or it won't work. * Try using some data with the Line chart graph template A thorough guide is needed, help is welcome!
== Documentation links == * Tabular help https://www.mediawiki.org/wiki/Help:Tabular_Data * Map help https://www.mediawiki.org/wiki/Help:Map_Data If you find a bug, create Phabricator ticket with #tabular-data tag, or comment on the documentation talk pages.
== FAQ == * Relation to Wikidata: Wikidata is about "facts" (small pieces of information). Structured data is about "blobs" - large amounts of data like the historical weather or the outline of the state of New York.
== TODOs == * Add a nice "table editor" - editing JSON by hand is cruel. T134618 * "What links here" should track data usage across wikis. Will allow quicker auto-refresh of the pages too. T153966 * Support data redirects. T153598 * Mega epic: Support external data feeds.
Great work!
I'm happy with the new naming, Commons Datasets.
For historical maps we have been waiting to have a way to store data about the rectification with the map image. Here it is! It brings us one notch closer to being able to work with zoomable historical maps in Wikipedia.
Some have noted that the datasets are contrary to what Wikidata is about. Instead, they are complementary! Not all data is either suitable for Wikidata or licensed openly enough. Or not yet. Many great datasets can be introduced to the Wikimedia online communities. The data owners will pay attention to more open licensing, seeing their data being used. The wikidatans will pick up interesting datasets and work to prepare missing properties for them in Wikidata and ignite their bots. Sometimes the data can just be used as is.
This is one part of the puzzle and it will be interesting to see how the pieces fall into their places and evolve further.
In the coming few days there'll be time to digest and experiment.
Happy holidays Susanna
2016-12-23 5:22 GMT+02:00 Bohdan Melnychuk base-w@yandex.ru:
Yay :)
As someone who already has plans to actively use it in both my metapedian role (e.g. CEE Spring article writing contest statistics data for building Graphs from being stored like https://meta.wikimedia.org/w/ index.php?title=User:BaseBot/CEES/MMXVI/Per_country_sums_( general)&action=edit https://meta.wikimedia.org/w/index.php?title=User:BaseBot/CEES/MMXVI/Per_country_sums_(general)&action=edit§ion=2 to a better format of https://commons.wikimedia.org/ wiki/Data:Wikimedia/CEE_Spring/Statistics/MMXVI/Per_ country_sums_(general).tab which can be turned by Lua to the same output but now with it being controlled on wiki rather than bot code part) and exopedianish for actual articles I think it is wonderful.
I do think that it needs tight cross linking with Wikidata and perfectly a way to run queries against both the sources at the same time (e.g. "give me the weather in all the current capitals in the date the comet Whatever was closest to the Sun the last time" or whatever else more useful thing may come into one's mind), but that does not deny the fact that it is very useful already.
It can also be used as an intermediate location for data on the way to be imported to Wikidata, IMHO.
--Base
22.12.2016, 21:31, "Yuri Astrakhan" yastrakhan@wikimedia.org:
Gift season! We have launched structured data on Commons, available from all wikis.
TLDR; One data store. Use everywhere. Upload table data to Commons, with localization, and use it to create wiki tables, lists, or use directly in graphs. Works for GeoJSON maps too. Must be licensed as CC0. Try this per-state GDP map demo, and select multiple years. More demos at the bottom. US Map state highlight https://en.wikipedia.org/wiki/Template:Graph:US_Map_state_highlight
Data can now be stored as *.tab and *.map pages in the data namespace on Commons. That data may contain localization, so a table cell could be in multiple languages. And that data is accessible from any wikis, by Lua scripts, Graphs, and Maps.
Lua lets you generate wiki tables from the data by filtering, converting, mixing, and formatting the raw data. Lua also lets you generate lists. Or any wiki markup.
Graphs can use both .tab and .map directly to visualize the data and let users interact with it. The GDP demo above uses a map from Commons, and colors each segment with the data based on a data table.
Kartographer (<maplink>/<mapframe>) can use the .map data as an extra layer on top of the base map. This way we can show endangered species' habitat.
== Demo ==
- Raw data example
https://commons.wikimedia.org/wiki/Data:Weather/New_York_City.tab
- Interactive Weather data
https://en.wikipedia.org/wiki/Template:Graph:Weather_monthly_history
- Same data in Weather template
https://en.wikipedia.org/wiki/User:Yurik/WeatherDemo
- Interactive GDP map
https://en.wikipedia.org/wiki/Template:Graph:US_Map_state_highlight
- Endangered Jemez Mountains salamander - habitat
https://en.wikipedia.org/wiki/Jemez_Mountains_salamander#/maplink/0
- Population history
https://en.wikipedia.org/wiki/Template:Graph:Population_history
== Getting started ==
- Try creating a page at data:Sandbox/<user>.tab on Commons. Don't forget
the .tab extension, or it won't work.
- Try using some data with the Line chart graph template
A thorough guide is needed, help is welcome!
== Documentation links ==
- Tabular help https://www.mediawiki.org/wiki/Help:Tabular_Data
- Map help https://www.mediawiki.org/wiki/Help:Map_Data
If you find a bug, create Phabricator ticket with #tabular-data tag, or comment on the documentation talk pages.
== FAQ ==
- Relation to Wikidata: Wikidata is about "facts" (small pieces of
information). Structured data is about "blobs" - large amounts of data like the historical weather or the outline of the state of New York.
== TODOs ==
- Add a nice "table editor" - editing JSON by hand is cruel. T134618
- "What links here" should track data usage across wikis. Will allow
quicker auto-refresh of the pages too. T153966
- Support data redirects. T153598
- Mega epic: Support external data feeds.
,
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
Speaking of historical maps,
The Warper has two core functions: - storing the control points - rectifying and cache/generate tiles
Moving the control points to Commons would make the Warper obsolete, because the map could be rectified client side(which makes sense anyway). Both .tab and .map could work.
We could just fetch both the image and the control points with JavaScript or have a Lua module to make the data more accessible and then just do the rendering with JavaScript.
It would not be to much work at all to get a UserScript working, but I'm hoping whatever we do this time could be compatible with future features in Kartographer.
Awesome work! // Albin
2016-12-23 9:19 GMT+01:00 Susanna Ånäs susanna.anas@gmail.com:
Great work!
I'm happy with the new naming, Commons Datasets.
For historical maps we have been waiting to have a way to store data about the rectification with the map image. Here it is! It brings us one notch closer to being able to work with zoomable historical maps in Wikipedia.
Some have noted that the datasets are contrary to what Wikidata is about. Instead, they are complementary! Not all data is either suitable for Wikidata or licensed openly enough. Or not yet. Many great datasets can be introduced to the Wikimedia online communities. The data owners will pay attention to more open licensing, seeing their data being used. The wikidatans will pick up interesting datasets and work to prepare missing properties for them in Wikidata and ignite their bots. Sometimes the data can just be used as is.
This is one part of the puzzle and it will be interesting to see how the pieces fall into their places and evolve further.
In the coming few days there'll be time to digest and experiment.
Happy holidays Susanna
2016-12-23 5:22 GMT+02:00 Bohdan Melnychuk base-w@yandex.ru:
Yay :)
As someone who already has plans to actively use it in both my metapedian role (e.g. CEE Spring article writing contest statistics data for building Graphs from being stored like https://meta.wikimedia.org/w/i ndex.php?title=User:BaseBot/CEES/MMXVI/Per_country_sums_(gen eral)&action=edit https://meta.wikimedia.org/w/index.php?title=User:BaseBot/CEES/MMXVI/Per_country_sums_(general)&action=edit§ion=2 to a better format of https://commons.wikimedia.org/ wiki/Data:Wikimedia/CEE_Spring/Statistics/MMXVI/Per_country_ sums_(general).tab which can be turned by Lua to the same output but now with it being controlled on wiki rather than bot code part) and exopedianish for actual articles I think it is wonderful.
I do think that it needs tight cross linking with Wikidata and perfectly a way to run queries against both the sources at the same time (e.g. "give me the weather in all the current capitals in the date the comet Whatever was closest to the Sun the last time" or whatever else more useful thing may come into one's mind), but that does not deny the fact that it is very useful already.
It can also be used as an intermediate location for data on the way to be imported to Wikidata, IMHO.
--Base
22.12.2016, 21:31, "Yuri Astrakhan" yastrakhan@wikimedia.org:
Gift season! We have launched structured data on Commons, available from all wikis.
TLDR; One data store. Use everywhere. Upload table data to Commons, with localization, and use it to create wiki tables, lists, or use directly in graphs. Works for GeoJSON maps too. Must be licensed as CC0. Try this per-state GDP map demo, and select multiple years. More demos at the bottom. US Map state highlight https://en.wikipedia.org/wiki/Template:Graph:US_Map_state_highlight
Data can now be stored as *.tab and *.map pages in the data namespace on Commons. That data may contain localization, so a table cell could be in multiple languages. And that data is accessible from any wikis, by Lua scripts, Graphs, and Maps.
Lua lets you generate wiki tables from the data by filtering, converting, mixing, and formatting the raw data. Lua also lets you generate lists. Or any wiki markup.
Graphs can use both .tab and .map directly to visualize the data and let users interact with it. The GDP demo above uses a map from Commons, and colors each segment with the data based on a data table.
Kartographer (<maplink>/<mapframe>) can use the .map data as an extra layer on top of the base map. This way we can show endangered species' habitat.
== Demo ==
- Raw data example
https://commons.wikimedia.org/wiki/Data:Weather/New_York_City.tab
- Interactive Weather data
https://en.wikipedia.org/wiki/Template:Graph:Weather_monthly_history
- Same data in Weather template
https://en.wikipedia.org/wiki/User:Yurik/WeatherDemo
- Interactive GDP map
https://en.wikipedia.org/wiki/Template:Graph:US_Map_state_highlight
- Endangered Jemez Mountains salamander - habitat
https://en.wikipedia.org/wiki/Jemez_Mountains_salamander#/maplink/0
- Population history
https://en.wikipedia.org/wiki/Template:Graph:Population_history
== Getting started ==
- Try creating a page at data:Sandbox/<user>.tab on Commons. Don't forget
the .tab extension, or it won't work.
- Try using some data with the Line chart graph template
A thorough guide is needed, help is welcome!
== Documentation links ==
- Tabular help https://www.mediawiki.org/wiki/Help:Tabular_Data
- Map help https://www.mediawiki.org/wiki/Help:Map_Data
If you find a bug, create Phabricator ticket with #tabular-data tag, or comment on the documentation talk pages.
== FAQ ==
- Relation to Wikidata: Wikidata is about "facts" (small pieces of
information). Structured data is about "blobs" - large amounts of data like the historical weather or the outline of the state of New York.
== TODOs ==
- Add a nice "table editor" - editing JSON by hand is cruel. T134618
- "What links here" should track data usage across wikis. Will allow
quicker auto-refresh of the pages too. T153966
- Support data redirects. T153598
- Mega epic: Support external data feeds.
,
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l