Has the format of the data lake history files changed? Some code I have which works with these assumes 70 fields, and that jives with what's documented at https://wikitech.wikimedia.org/wiki/Data_Platform/Data_Lake/Edits/MediaWiki_.... But it looks like there's now 76 fields:
$ bzcat 2026-01.enwiki.2026-01.tsv.bz2 | head -1 | tr '\t' '\n' | wc -l 76
yes, it appears that changes like this one https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/refinery/+/67099aa6850d136f139e474c9429a14447a99ccd%5E%21/#F2 will automatically be picked up by the job, because the dumps job https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/refinery/source/+/refs/heads/master/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/mediawikihistory/MediawikiHistoryDumper.scala doesn't specify a set of fields. We'll have to make the job more explicit and communicate changes like this better. It does look like Isaac updated the schema definition https://wikitech.wikimedia.org/w/index.php?title=Data_Platform%2FData_Lake%2FEdits%2FMediaWiki_history_dumps&diff=2372750&oldid=2281583 on Wikitech, thank you Isaac!
On Sun, Mar 15, 2026 at 3:38 PM Roy Smith via Analytics < analytics@lists.wikimedia.org> wrote:
Has the format of the data lake history files changed? Some code I have which works with these assumes 70 fields, and that jives with what's documented at https://wikitech.wikimedia.org/wiki/Data_Platform/Data_Lake/Edits/MediaWiki_.... But it looks like there's now 76 fields:
$ bzcat 2026-01.enwiki.2026-01.tsv.bz2 | head -1 | tr '\t' '\n' | wc -l
76
Analytics mailing list -- analytics@lists.wikimedia.org To unsubscribe send an email to analytics-leave@lists.wikimedia.org
So where can I find the current schema definition?
On Mar 16, 2026, at 6:46 AM, Dan Andreescu via Analytics analytics@lists.wikimedia.org wrote:
yes, it appears that changes like this one https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/refinery/+/67099aa6850d136f139e474c9429a14447a99ccd%5E%21/#F2 will automatically be picked up by the job, because the dumps job https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/refinery/source/+/refs/heads/master/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/mediawikihistory/MediawikiHistoryDumper.scala doesn't specify a set of fields. We'll have to make the job more explicit and communicate changes like this better. It does look like Isaac updated the schema definition https://wikitech.wikimedia.org/w/index.php?title=Data_Platform%2FData_Lake%2FEdits%2FMediaWiki_history_dumps&diff=2372750&oldid=2281583 on Wikitech, thank you Isaac!
On Sun, Mar 15, 2026 at 3:38 PM Roy Smith via Analytics <analytics@lists.wikimedia.org mailto:analytics@lists.wikimedia.org> wrote: Has the format of the data lake history files changed? Some code I have which works with these assumes 70 fields, and that jives with what's documented at https://wikitech.wikimedia.org/wiki/Data_Platform/Data_Lake/Edits/MediaWiki_... https://wikitech.wikimedia.org/wiki/Data_Platform/Data_Lake/Edits/MediaWiki_history_dumps#Important_fields:_event_entity_and_event_type. But it looks like there's now 76 fields:
$ bzcat 2026-01.enwiki.2026-01.tsv.bz2 | head -1 | tr '\t' '\n' | wc -l 76
Analytics mailing list -- analytics@lists.wikimedia.org mailto:analytics@lists.wikimedia.org To unsubscribe send an email to analytics-leave@lists.wikimedia.org mailto:analytics-leave@lists.wikimedia.org _______________________________________________ Analytics mailing list -- analytics@lists.wikimedia.org To unsubscribe send an email to analytics-leave@lists.wikimedia.org
https://wikitech.wikimedia.org/wiki/Data_Platform/Data_Lake/Edits/MediaWiki_...
On Mon, Mar 16, 2026 at 8:02 AM Roy Smith roy@panix.com wrote:
So where can I find the current schema definition?
On Mar 16, 2026, at 6:46 AM, Dan Andreescu via Analytics < analytics@lists.wikimedia.org> wrote:
yes, it appears that changes like this one https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/refinery/+/67099aa6850d136f139e474c9429a14447a99ccd%5E%21/#F2 will automatically be picked up by the job, because the dumps job https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/refinery/source/+/refs/heads/master/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/mediawikihistory/MediawikiHistoryDumper.scala doesn't specify a set of fields. We'll have to make the job more explicit and communicate changes like this better. It does look like Isaac updated the schema definition https://wikitech.wikimedia.org/w/index.php?title=Data_Platform%2FData_Lake%2FEdits%2FMediaWiki_history_dumps&diff=2372750&oldid=2281583 on Wikitech, thank you Isaac!
On Sun, Mar 15, 2026 at 3:38 PM Roy Smith via Analytics < analytics@lists.wikimedia.org> wrote:
Has the format of the data lake history files changed? Some code I have which works with these assumes 70 fields, and that jives with what's documented at https://wikitech.wikimedia.org/wiki/Data_Platform/Data_Lake/Edits/MediaWiki_.... But it looks like there's now 76 fields:
$ bzcat 2026-01.enwiki.2026-01.tsv.bz2 | head -1 | tr '\t' '\n' | wc -l 76
Analytics mailing list -- analytics@lists.wikimedia.org To unsubscribe send an email to analytics-leave@lists.wikimedia.org
Analytics mailing list -- analytics@lists.wikimedia.org To unsubscribe send an email to analytics-leave@lists.wikimedia.org
Not an official definition, but you're also welcome to use my schema in this PAWS notebook (which is what Dan linked but in Python code): https://public-paws.wmcloud.org/User:Isaac_(WMF)/mediawiki-history-dumps-exa...
Good luck with Mediawiki History -- one of my favorite datasets out there and one day at a hackathon, I'll see about making a Python library for it.
On Mon, Mar 16, 2026 at 8:32 AM Dan Andreescu via Analytics < analytics@lists.wikimedia.org> wrote:
https://wikitech.wikimedia.org/wiki/Data_Platform/Data_Lake/Edits/MediaWiki_...
On Mon, Mar 16, 2026 at 8:02 AM Roy Smith roy@panix.com wrote:
So where can I find the current schema definition?
On Mar 16, 2026, at 6:46 AM, Dan Andreescu via Analytics < analytics@lists.wikimedia.org> wrote:
yes, it appears that changes like this one https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/refinery/+/67099aa6850d136f139e474c9429a14447a99ccd%5E%21/#F2 will automatically be picked up by the job, because the dumps job https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/refinery/source/+/refs/heads/master/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/mediawikihistory/MediawikiHistoryDumper.scala doesn't specify a set of fields. We'll have to make the job more explicit and communicate changes like this better. It does look like Isaac updated the schema definition https://wikitech.wikimedia.org/w/index.php?title=Data_Platform%2FData_Lake%2FEdits%2FMediaWiki_history_dumps&diff=2372750&oldid=2281583 on Wikitech, thank you Isaac!
On Sun, Mar 15, 2026 at 3:38 PM Roy Smith via Analytics < analytics@lists.wikimedia.org> wrote:
Has the format of the data lake history files changed? Some code I have which works with these assumes 70 fields, and that jives with what's documented at https://wikitech.wikimedia.org/wiki/Data_Platform/Data_Lake/Edits/MediaWiki_.... But it looks like there's now 76 fields:
$ bzcat 2026-01.enwiki.2026-01.tsv.bz2 | head -1 | tr '\t' '\n' | wc -l 76
Analytics mailing list -- analytics@lists.wikimedia.org To unsubscribe send an email to analytics-leave@lists.wikimedia.org
Analytics mailing list -- analytics@lists.wikimedia.org To unsubscribe send an email to analytics-leave@lists.wikimedia.org
Analytics mailing list -- analytics@lists.wikimedia.org To unsubscribe send an email to analytics-leave@lists.wikimedia.org
Ah! Apologies! We did this for the upstream internal mediawiki_history dataset https://wikitech.wikimedia.org/wiki/Data_Platform/Data_Lake/Edits/MediaWiki_history#Changes_and_known_problems, but did not consider the downstream public dump files.
I've added an entry in the Changes and Known problems section of the upstream mediawiki_history docs https://wikitech.wikimedia.org/w/index.php?title=Data_Platform%2FData_Lake%2FEdits%2FMediaWiki_history&diff=2391314&oldid=2320784, and added a link to that section in the dumps files docs too https://wikitech.wikimedia.org/w/index.php?title=Data_Platform%2FData_Lake%2FEdits%2FMediaWiki_history_dumps&diff=2391315&oldid=2372750 .
See also T365648 Add user_central_id to mediawiki_history and mediawiki_history_reduced Hive tables https://phabricator.wikimedia.org/T365648
On Mon, Mar 16, 2026 at 8:42 AM Isaac Johnson via Analytics < analytics@lists.wikimedia.org> wrote:
Not an official definition, but you're also welcome to use my schema in this PAWS notebook (which is what Dan linked but in Python code): https://public-paws.wmcloud.org/User:Isaac_(WMF)/mediawiki-history-dumps-exa...
Good luck with Mediawiki History -- one of my favorite datasets out there and one day at a hackathon, I'll see about making a Python library for it.
On Mon, Mar 16, 2026 at 8:32 AM Dan Andreescu via Analytics < analytics@lists.wikimedia.org> wrote:
https://wikitech.wikimedia.org/wiki/Data_Platform/Data_Lake/Edits/MediaWiki_...
On Mon, Mar 16, 2026 at 8:02 AM Roy Smith roy@panix.com wrote:
So where can I find the current schema definition?
On Mar 16, 2026, at 6:46 AM, Dan Andreescu via Analytics < analytics@lists.wikimedia.org> wrote:
yes, it appears that changes like this one https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/refinery/+/67099aa6850d136f139e474c9429a14447a99ccd%5E%21/#F2 will automatically be picked up by the job, because the dumps job https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/refinery/source/+/refs/heads/master/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/mediawikihistory/MediawikiHistoryDumper.scala doesn't specify a set of fields. We'll have to make the job more explicit and communicate changes like this better. It does look like Isaac updated the schema definition https://wikitech.wikimedia.org/w/index.php?title=Data_Platform%2FData_Lake%2FEdits%2FMediaWiki_history_dumps&diff=2372750&oldid=2281583 on Wikitech, thank you Isaac!
On Sun, Mar 15, 2026 at 3:38 PM Roy Smith via Analytics < analytics@lists.wikimedia.org> wrote:
Has the format of the data lake history files changed? Some code I have which works with these assumes 70 fields, and that jives with what's documented at https://wikitech.wikimedia.org/wiki/Data_Platform/Data_Lake/Edits/MediaWiki_.... But it looks like there's now 76 fields:
$ bzcat 2026-01.enwiki.2026-01.tsv.bz2 | head -1 | tr '\t' '\n' | wc -l 76
Analytics mailing list -- analytics@lists.wikimedia.org To unsubscribe send an email to analytics-leave@lists.wikimedia.org
Analytics mailing list -- analytics@lists.wikimedia.org To unsubscribe send an email to analytics-leave@lists.wikimedia.org
Analytics mailing list -- analytics@lists.wikimedia.org To unsubscribe send an email to analytics-leave@lists.wikimedia.org
-- Isaac Johnson https://meta.wikimedia.org/wiki/User:Isaac_(WMF) (he/him) -- Senior Research Scientist -- Wikimedia Foundation _______________________________________________ Analytics mailing list -- analytics@lists.wikimedia.org To unsubscribe send an email to analytics-leave@lists.wikimedia.org