Lars Aronsson wrote:
Are visitor stats (as produced by Domas) safely archived somewhere, for example on the toolserver, where development projects can easily access them for analysis? I have made my own copies of the files (I guess my plan was to use them, but this hasn't started yet), but now I'm running out of disk and I urgently need to clear some space on that server.
I just deleted September 2009 (last 2 weeks) and that freed 9 GB.
The oldest I have is pagecounts-20071209-180000.gz
As Platonides mentioned, they are in /mnt/user-store/stats on the toolserver; however, I would not call that "safely archived": one of my cron jobs just copies them from Domas server, and that's it.
At the moment, there should be everything starting from 1 January 2009 (although part of it disappeared at some point, but I managed to recover it).
However, this is definitively not a sustainable solution in the long run: the files currently take 335 Gb (out of a 1.5 Tb total space).
Erik Zachte stores archives of visitor stats in a better format, aggregating some of the older data and storing several days of data in one file. I started looking into these files earlier this year, planning to spend some time playing with this data. One of my ideas was to replicate the statistical data that is on the WMF stats server somewhere on the toolserver -- and do it "officially" and not just by copying files using a personal cron job. Unfortunately, "real life" took over and I did not manage to continue this (and still can't). However, if there is any interest in improving the situation, I'd be glad to look into it as soon as I can.
I cc' Erik who may have more to say.
Cheers,
Frédéric