On Sat, Oct 11, 2008 at 6:30 PM, Chad innocentkiller@gmail.com wrote:
On Sat, Oct 11, 2008 at 10:26 AM, Thomas Dalton thomas.dalton@gmail.comwrote:
Indeed, you can make some dumps more frequent at the expense of making others less frequent. No-one has yet explained why small dumps should be more frequent that large ones.
Or, for that matter, what is gained by more frequent dumps, period. 6 weeks isn't a massive amount of time...
on en,wikt, we have several dozen reports and such that need updating to manage a lot of details. 6 weeks is *interminable*. Which is why we are running daily incrementals now
Why should small dumps be more frequent than large ones? Because they should be weekly. The problem is that the large ones take much too long, and clog the queue.
This is not rocket science people, it just needs one thread that doesn't get blocked. Simple to do. In the loop in findAndLockNextWiki (for db), do
if '--small' in sys.argv[1:] and db.description() in ['enwiki', 'dewiki', 'frwiki', 'plwiki', 'jawiki, 'itwiki', 'nlwiki', 'ptwiki', 'eswiki', 'ruwiki']: continue
that is all. Then run one thread with --small.
(those are the 10 largest pedias, the ones with more than 10M edits)