jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/375448 )
Change subject: Make _flush aware of _putthread ongoing tasks ......................................................................
Make _flush aware of _putthread ongoing tasks
Make sure _flush() is aware if _putthread() is idle or working on a task. Relying on page_put_queue.qsize() > 0 only, as a condition to keep the while loop alive is not safe as the last item in the queue can be fetched by async_manager() before page_put_queue.qsize() is checked and then found empty. In this case _flush() returns before all pending requests are served.
A new Queue is introduced as a form of synchronisation between MainThread in _flush() and asyc_manager(), signalling when async_manager is actually working and not idle waiting for a request.
Bug: T147178 Change-Id: I184b445c43aa44cb000bfacc1cad18d2de265c48 --- M pywikibot/__init__.py 1 file changed, 7 insertions(+), 1 deletion(-)
Approvals: Zhuyifei1999: Looks good to me, approved jenkins-bot: Verified
diff --git a/pywikibot/__init__.py b/pywikibot/__init__.py index 9c9632d..6fe82cc 100644 --- a/pywikibot/__init__.py +++ b/pywikibot/__init__.py @@ -1372,7 +1372,9 @@ '{lightblue}Waiting for {num} pages to be put. ' 'Estimated time remaining: {sec}{default}', num=num, sec=sec))
- while _putthread.is_alive() and page_put_queue.qsize() > 0: + while (_putthread.is_alive() + and (page_put_queue.qsize() > 0 + or page_put_queue_busy.qsize() > 0)): try: _putthread.join(1) except KeyboardInterrupt: @@ -1398,10 +1400,12 @@ """Daemon; take requests from the queue and execute them in background.""" while True: (request, args, kwargs) = page_put_queue.get() + page_put_queue_busy.put(None) if request is None: break request(*args, **kwargs) page_put_queue.task_done() + page_put_queue_busy.get()
def async_request(request, *args, **kwargs): @@ -1420,6 +1424,8 @@
# queue to hold pending requests page_put_queue = Queue(config.max_queue_size) +# queue to signal that async_manager is working on a request. See T147178. +page_put_queue_busy = Queue(config.max_queue_size) # set up the background thread _putthread = threading.Thread(target=async_manager) # identification for debugging purposes