jenkins-bot merged this change.

View Change

Make _flush aware of _putthread ongoing tasks

Make sure _flush() is aware if _putthread() is idle or working on a
task.
Relying on page_put_queue.qsize() > 0 only, as a condition to keep the while
loop alive is not safe as the last item in the queue can be fetched by
async_manager() before page_put_queue.qsize() is checked and then found empty.
In this case _flush() returns before all pending requests are served.

A new Queue is introduced as a form of synchronisation between
MainThread in _flush() and asyc_manager(), signalling when async_manager
is actually working and not idle waiting for a request.

Bug: T147178
Change-Id: I184b445c43aa44cb000bfacc1cad18d2de265c48
---
M pywikibot/__init__.py
1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/pywikibot/__init__.py b/pywikibot/__init__.py
index 9c9632d..6fe82cc 100644
--- a/pywikibot/__init__.py
+++ b/pywikibot/__init__.py
@@ -1372,7 +1372,9 @@
             '{lightblue}Waiting for {num} pages to be put. '
             'Estimated time remaining: {sec}{default}', num=num, sec=sec))
 
-    while _putthread.is_alive() and page_put_queue.qsize() > 0:
+    while (_putthread.is_alive()
+           and (page_put_queue.qsize() > 0
+                or page_put_queue_busy.qsize() > 0)):
         try:
             _putthread.join(1)
         except KeyboardInterrupt:
@@ -1398,10 +1400,12 @@
     """Daemon; take requests from the queue and execute them in background."""
     while True:
         (request, args, kwargs) = page_put_queue.get()
+        page_put_queue_busy.put(None)
         if request is None:
             break
         request(*args, **kwargs)
         page_put_queue.task_done()
+        page_put_queue_busy.get()
 
 
 def async_request(request, *args, **kwargs):
@@ -1420,6 +1424,8 @@
 
 # queue to hold pending requests
 page_put_queue = Queue(config.max_queue_size)
+# queue to signal that async_manager is working on a request. See T147178.
+page_put_queue_busy = Queue(config.max_queue_size)
 # set up the background thread
 _putthread = threading.Thread(target=async_manager)
 # identification for debugging purposes

To view, visit change 375448. To unsubscribe, or for help writing mail filters, visit settings.