Am 10.12.2011 20:52, schrieb Jeremy Baron:
Is it sufficient to receive the XML on stdin or do you
need to be able to seek?
It is trivial to give you XML on stdin e.g.
$< path/to/bz2 bzip2 -d | perl script.pl
Hmm, the stdin is possible, but I think this will need many memory of
RAM on the server. I think this is no option for the future. Every
language grows every day and the dumps will also grow. The next problem
is the parallel use of a compressed file. If more user use this
compressed file like your idea, then bzip2 will crash the server IMHO.
I think it is no problem to store the uncompressed XML files for an easy
usage. We should make rules, where they have to stay and how long or we
need a list, where every user can say "I need only the two newest dumps
of enwiki, dewiki,...". If a dump is not needed, then we can delete this
file.
Stefan (sk)