Sj:
> What is it that will take a lot of time to update, rewriting the queries?
> There are some SQL-friendly people around here who would like to help with
that.
I know SQL. Right now the scripts are parsing the dumps directly, doing a
lot of encoding, splitting, sorting and merging to process the files within
cramped memory.
SQL would be much easier. I do however worry about run time. I asked Brion
at Berlin, and he had no idea either.
Heavy ad hoc SQL queries have already often been forcefully aborted or
forbidden alltogether, no doubt for good reasons, so what to expect for this
monster job? It already runs 10 hours on English Wikipedia alone, over 24
hours for all Wikipedias. And parsing the dumps should still be much faster
than SQL, with all its extra I/O. Of course some tests could shed light on
this.
Apart from that I do lack the energy for any serious programming. All that
is left is for my day job. So I won't work on migrating to SQL of write new
encoding, splitting, sorting and merging code anytime soon. If I do some
perl hacking during my holiday it should be targeted towards EasyTimeline
where I have made promises that are half a year overdue (mainly unicode
support). If someone else would want to step in for wikistats code, that
would of course be great.
Erik Zachte