https://bugzilla.wikimedia.org/show_bug.cgi?id=55374
Web browser: --- Bug ID: 55374 Summary: very slow interwiki.py and urlopen error with disambiguations Product: Pywikibot Version: compat (1.0) Hardware: All OS: All Status: NEW Severity: normal Priority: Unprioritized Component: interwiki.py Assignee: Pywikipedia-bugs@lists.wikimedia.org Reporter: jan.dudik@gmail.com Classification: Unclassified Mobile Platform: ---
After change in disambiguationpages is interwiki bot very very slow. Loading 60 pages from source language takes more than one minute (from other languages is time standard).
Additionally bot often (sometimes every minute, sometimes after 10 minutes) frozes because of no response from server, see log:
interwiki.py -new -family:wiktionary -wiktionary -async -autonomous -cleanup
Getting 60 pages from wiktionary:cs... NOTE: [[cs:synowa]] does not have any interwiki links ... ERROR: URLError: <urlopen error [Errno 10060] ...> WARNING: Could not open 'https://cs.wiktionary.org/w/api.php?action=query&format =json&titles=steteramus&ppprop=disambiguation&prop=pageprops'. Maybe the server or your connection is down. Retrying in 1 minutes...
https://bugzilla.wikimedia.org/show_bug.cgi?id=55374
Malafaya malafaya@clix.pt changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |malafaya@clix.pt
--- Comment #1 from Malafaya malafaya@clix.pt --- That seems because for each existing page it makes an API request for its disambiguation property. I wonder if this couldn't be done in the first page request when checking for its existence, insteand of making an additional query.
https://bugzilla.wikimedia.org/show_bug.cgi?id=55374
Amir Ladsgroup ladsgroup@gmail.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Priority|Unprioritized |Normal Status|NEW |RESOLVED CC| |jan.dudik@gmail.com, | |ladsgroup@gmail.com Resolution|--- |FIXED
--- Comment #2 from Amir Ladsgroup ladsgroup@gmail.com --- I checked and even with very slow internet of mine (100KB/s) It's working fast and correct, system of checking disambiguation has changed so I close this bug as fixed but If It's still really slow, feel free to reopen it
https://bugzilla.wikimedia.org/show_bug.cgi?id=55374
Malafaya malafaya@clix.pt changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED Resolution|FIXED |---
--- Comment #3 from Malafaya malafaya@clix.pt --- I'm not sure it's faster, just it looks different. It seems to take longer in the beginning now (loading first 50 pages from Wiktionary takes 1 minute or so). It seems to be checking for the disambiguation page attribute individually when it fetches the page titles, instead of then it processes the page. That would be the same problem. But again, that's just my feeling, not that I examined the code. Either way, it's still slow, just in a different phase of the process.
https://bugzilla.wikimedia.org/show_bug.cgi?id=55374
JAn Dudík jan.dudik@gmail.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|REOPENED |RESOLVED Resolution|--- |WORKSFORME
pywikipedia-bugs@lists.wikimedia.org