Hello all,
I need help. I have a perl programme (for Check Wikipedia). This can scan a dump of a language very fast. 2200 pages per minute is no problem.
I will daily scan with the same script the page text of the live Wikipedia. Not all pages, but maybe 20000 per day per language. Normally this need only 10 minutes with the dump, but with the live Wikipedia this need many time. I use the Wikipedia-API to get the text of an article and so my script can only scan 120 pages per minute. So this scan need at the moment in enwiki 300 minutes or in dewiki 134 minutes. The most time my script is waiting. This is a problem because there is a high CPU usage.
I need a faster way to get the text from the live Wikipedia. So I can reduce the CPU usage.
Maybe someone know a faster way! Or have an other idea.
Thanks for every help, Stefan (sk)
More Info: http://de.wikipedia.org/wiki/Benutzer:Stefan_K%C3%BChn/Check_Wikipedia http://en.wikipedia.org/wiki/Wikipedia:WikiProject_Check_Wikipedia http://toolserver.org/~sk/checkwiki/checkwiki.pl