[Toolserver-l] Problem

[Stefan Kühn]

Hello all,

I need help. I have a perl programme (for Check Wikipedia). This can 
scan a dump of a language very fast. 2200 pages per minute is no problem.

I will daily scan with the same script the page text of the live 
Wikipedia. Not all pages, but maybe 20000 per day per language. Normally 
this need only 10 minutes with the dump, but with the live Wikipedia 
this need many time. I use the Wikipedia-API to get the text of an 
article and so my script can only scan 120 pages per minute. So this 
scan need at the moment in enwiki 300 minutes or in dewiki 134 minutes. 
The most time my script is waiting. This is a problem because there is a 
high CPU usage.

I need a faster way to get the text from the live Wikipedia. So I can 
reduce the CPU usage.

Maybe someone know a faster way! Or have an other idea.

Thanks for every help,
Stefan (sk)


More Info:
http://de.wikipedia.org/wiki/Benutzer:Stefan_Kühn/Check_Wikipedia
http://en.wikipedia.org/wiki/Wikipedia:WikiProject_Check_Wikipedia
http://toolserver.org/~sk/checkwiki/checkwiki.pl