Yes, working with large data sets is fun. There are always surprises. For example top of the 'words, most reverted by trusted users' are not the expected infantile type of things either. I haven't done the analysis on the full dump yet, but on the subset from the full histories of articles from the PAN 10 LAB test set following words came up top (sorted by chi-square, note that this is very preliminary and tokenization/regularization might have been wrong): token, chi-sq, regular-diff-tok-cnt, revert-diff-tok-cnt Image:Example.jpg|Ca 87959701.9043 113 7568 [[Media:Example.ogg] 5549492.56549 62 2196 title]][http://www.e 606182.771025 0 363 aaaaaaaaaaaaaaaaaaaa 305908.640902 0 365 [http://youtube.com/ 253267.5237 189 407 pooooooooooooooooooo 214921.014803 0 375 you 154597.596739 18655 102007 ffffffffffffffffffff 129822.419517 1 238 value="transparent"> 129575.702482 1 168 ____________________ 126503.143626 23 166 language|Macedonian] 123467.452157 121 164 hhhhhhhhhhhhhhhhhhhh 119613.359035 0 280 !!!!!!!!!!!!!!!!!!!! 118479.373501 5 686 AAAAAAAAAAAAAAAAAAAA 114581.582068 2 158 oooooooooooooooooooo 110263.074451 0 155 i 109590.406785 2620 55971 -- Cheers, Dmitry On Fri, Aug 13, 2010 at 4:06 PM, Luca de Alfaro <luca(a)dealfaro.org> wrote:

Yes. It is fairly easy to produce the list limited to a time period, or any other custom stats (e.g. 'reverted edits ratios' for anonymous users, etc). It's just several hours of processing. But it is limited with the time frame of the recent database dump. For the en-wiki it is 2010/01/30. Send your complains to the xmldatadumps-l (xmldatadumps-l(a)lists.wikimedia.org) ;) . By the way, I've posted (somewhat cleaned-up) python script that I've used to calculate that list. It's available here: http://code.google.com/p/pymwdat/ For en-wiki dump requires: * 31 Gb enwiki-20100130-pages-meta-history.xml.7z download; * 250 Gb free disk space (for intermediate data & dump); * ~week to pre-process the dump (modern desktop); * ~3 hours to do a simple run (e.g calculate the list like I did). Dump preprocessed is basically extracting/parsing .xml.7z, calculating MD5s for page revisions, calculating page diffs and pickling the results (alongside with other metadata) to disk. It uses a custom diff algorithm optimized for the wikipedia (regular diff is a way too slow and doesn't handle copy editing well). It needs memory if one wants to calculate/hold stats for every editor/page (4Gb minimal, 8Gb recommended, 24Gb+ preferred). But obviously one can filter yourself a data subset or even work on a single page. Requires System/Libraries: * Python 2.6+, Linux (I've never tried it on Windows); * PyWikipedia/Trunk ( http://svn.wikimedia.org/svnroot/pywikipedia/trunk/pywikipedia/ ) * OrderedDict (available in Python 2.7 or http://pypi.python.org/pypi/ordereddict/) * 7-Zip (command line 7za) -- Dmitry On Thu, Aug 19, 2010 at 8:46 AM, John Vandenberg <jayvdb(a)gmail.com> wrote: