[Pywikipedia-l] [ pywikipediabot-Bugs-1973804 ] Huge memory consumption during changing process

[SourceForge.net]

Bugs item #1973804, was opened at 2008-05-27 02:53
Message generated for change (Comment added) made by melancholie
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=603138&aid=1973804&group_id=93107

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: interwiki
Group: None
Status: Open
Resolution: None
Priority: 8
Private: No
Submitted By: Melancholie (melancholie)
Assigned to: Nobody/Anonymous (nobody)
Summary: Huge memory consumption during changing process

Initial Comment:
As soon as the changing process (putting/saving of pages) is started, interwiki.py (r5440) consumes more than 100 MB of memory (RAM+Swap) if bot is working on many wikis. Memory usage grows during changing process. When changing process is finished, the memory suddenly gets flushed. Memory usage is normal again then, but only until the next 'putting-pages process' proceeds ;-)

----------------------------------------------------------------------

>Comment By: Melancholie (melancholie)
Date: 2008-05-29 09:09

Message:
Logged In: YES 
user_id=2089773
Originator: YES

This bug is definitely because of that change:
http://svn.wikimedia.org/viewvc/pywikipedia/trunk/pywikipedia/wikipedia.py?r1=5437&r2=5438

----------------------------------------------------------------------

Comment By: Melancholie (melancholie)
Date: 2008-05-28 07:21

Message:
Logged In: YES 
user_id=2089773
Originator: YES

On low memory systems that does even lead to:
Inconsistency detected by ld.so: dl-minimal.c: 84: __libc_memalign:
Assertion `page != ((void *) -1)' failed!

Does that have to do with BeautifulSoup.py?
The revision that used (c)ElementTree did not cause that kind of bug!

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=603138&aid=1973804&group_id=93107