Hello gurus,
we (Kolossos and I) work an a new update for Wikipedia-World and
Vorlagenauswertung (Templatetiger)
http://de.wikipedia.org/wiki/Wikipedia:WikiProjekt_Georeferenzierung/Wikipe…
http://de.wikipedia.org/wiki/Wikipedia:WikiProjekt_Vorlagenauswertung/en
In the last years we use the dump from Wikimedia Foundation. We download
this dump for the main languages and unpack this big files at the
toolserver and scan this file with a perl-script for geocoordinates and
templates.
Problem: This need many time, make many traffic and need many disk space
at the toolserver. Also the update is only possible if a new dump is
available.
Now we want create a new workflow. We use a up-to-date list of articles
of a language and the perl-script get the text of a any article from the
toolserver.
It work very well, but we need at the moment for reading the text of an
article 2 seconds. If we get all 677000 German articles with 2 seconds
we need 15.6 days. This is too much! With the old workflow I need not
more then 30 Minutes to scan the complete DE-Dump.
I try different ways to get the text of an article with my perl-script.
1.)
http://localhost/~daniel/WikiSense/WikiProxy.php?wiki=de.wikipedia.org&…
2.)
http://tools.wikimedia.de/~kolossos/rss-test/fopen.php?pro=wikipedia&la…
but every time the perl-script need average 2 seconds. With the second
PHP-Script I see in a browser the time for getting the text and this
time is mostly 0.2 seconds (factor 10 lower !!!) In this time we need
only 1.5 days for scanning. This will be ok!
Here is the perl-code of my script, which get the article test. I hope
one perl guru can help us to reduce the time.
Thank´s for every help!
Stefan Kühn
sub get_article_text_from_web {
my $title = $_[0];
my $page_id = 0;
my $revision_id = 0;
my $revision_time = 0;
my $text = "";
print "get\t".$title."\n";
my $test1 = time();
print localtime($test1)."\n";
use URI::Escape;
use LWP::UserAgent;
#
http://localhost/~daniel/WikiSense/WikiProxy.php?wiki=$lang.wikipedia.org&a…
my $url =
'http://localhost/~daniel/WikiSense/WikiProxy.php?wiki=de.wikipedia.org&title='.$title;
#http://www.webkuehn.de
uri_escape($url);
my $ua = LWP::UserAgent->new;
my $response = $ua->get( $url );
$response->is_success or die "$url: ", $response->status_line;
my $result = $response->content;
if ( $result ) {
$text = $result;
#print "$result\n";
print "ok\t".$title."\n";
my $test2 = time();
my $test3 = $test2 - $test1;
print localtime($test2)."\n";
print "second\t".$test3."\n\n";
} else {
print "No result $title\n";
}
#print "ok2\n";
return($title, $page_id, $revision_id, $revision_time, $text);
}