Thanks.Now, there is no sleep time between writes. It will reduce our
deployment time to a large extent.
I tried several hacks to remove sleep time without success. I didnt imagine
such a simple fix would settle this. I believed i was completely familiar
with pywikipedia framework but i guess there is still a lot to know. I
noticed several threads to improve pywikipedia framework documentation.I
strongly agree now. :)
Thanks,
Chinmay
On Wed, Aug 21, 2013 at 12:27 AM, Merlijn van Deen <valhallasw(a)arctus.nl>wrote;wrote:
The sleep time as implemented in pywikipedia is
influenced by three
factors:
- the minimum time between requests, minthrottle=1 (by default) in
user-config.py
- the minimum time between page saves, put_throttle=10
- the database lag on the DB servers - pages will not be saved unless the
lag is less than maxlag = 5
(all values in seconds).
In the case of wikidata, the maximum number of saves/edits per minute is
60 (iirc), so you could reduce put_throttle to 1 by adding
put_throttle=1
in your user-config.py
Best,
Merlijn
On 20 August 2013 20:16, Chinmay Naik <chin.naik26(a)gmail.com> wrote:
Hi,
I am Chinmay Naik, a Gsoc intern and operator of ProteinBoxBot(runs on
pywikipedia - formerly rewrite branch). When i run the bot, there exists a
sleep time of around 10 s (7s average) between two successive writes.
The bot will handle around 40,000 wikidata items corresponding to gene
wiki articles from
http://en.wikipedia.org/wiki/Category:Human_proteins. Also , i will
be uploading around 30 (claims + sources) for each
wikidata item.
If u take a look at the recent edits (
https://www.wikidata.org/w/index.php?title=Special:Contributions/ProteinBox…)oteinBoxBot),
it takes around 5 mins to completely write all claims+sources to each
wikidata item. This is a very large time lag and will hugely affect
deployment. Is there any way to reduce this sleep time? Any pointers on
this would be helpful.
Thanks,
Chinmay
_______________________________________________
Pywikipedia-l mailing list
Pywikipedia-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l
_______________________________________________
Pywikipedia-l mailing list
Pywikipedia-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l