As part of Wikimedia Germany's work around reference reuse, we wrote a
tool which processes the HTML dumps of all articles and produces
detailed information about how Cite references (and Kartographer maps)
are used on each page.
I'm writing this list for advice on how to publish the results so that
the data can be easily discovered and consumed by researchers.
Currently, the data is contained in 3,100 JSON and NDJSON files hosted
on a Wikimedia Cloud VPS server, with a total size of 3.4GB. The
outputs can be split or merged into whatever form will make them more
useable.
For an overview of the columns and sample rows, please see this task:
https://phabricator.wikimedia.org/T341751
We plan to run the scraper again in the future, and its modular
architecture makes it simple to include or exclude additional
information if anyone has suggestions about what else we might want to
extract from rendered articles. To read more about the tool itself and
why we decided to process HTML dumps directly, see this post:
https://mw.ludd.net/wiki/Elixir/HTML_dump_scraper
-Adam Wight
[[mw:Adamw]]
https://meta.wikimedia.org/wiki/WMDE_Technical_Wishes