On 10/01/11 22:13, Ariel T. Glenn wrote:
So "soon" took longer than I would have
liked. However, we are up and
running with the new code. I have started a few processes going and
over the next few days I will ramp it up to the usual number. In
particular I want to start a separate job for the larger wikis so that
the smaller jobs don't get trapped behind them.
Guess I'd better go update the various pages on wikitech now.
Thanks Ariel, that's good to hear.
Would it be possible to take this a step further, and for a single job
to be started up just for enwiki?
enwiki is unique among all the dumps in that it is the only one that
regularly fails more often than it succeeds; even partial dumps are
better than none, and enwiki also takes longer than any other dump
before it (typically) fails, so retrying it more aggressively than
others -- and independently of them, so it does not hold the other wikis
up -- would seem appropriate.
Thus, under this proposal, there would be three jobs running:
* other large wikis
* all small wikis
Ah yes, sorry that wasn't clear from the earlier message. I already
pulled enwiki oout of the main list and it will run as a bunch of
smaller parallel jobs on its own host.