jenkins-bot has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/375448 )
Change subject: Make _flush aware of _putthread ongoing tasks
......................................................................
Make _flush aware of _putthread ongoing tasks
Make sure _flush() is aware if _putthread() is idle or working on a
task.
Relying on page_put_queue.qsize() > 0 only, as a condition to keep the while
loop alive is not safe as the last item in the queue can be fetched by
async_manager() before page_put_queue.qsize() is checked and then found empty.
In this case _flush() returns before all pending requests are served.
A new Queue is introduced as a form of synchronisation between
MainThread in _flush() and asyc_manager(), signalling when async_manager
is actually working and not idle waiting for a request.
Bug: T147178
Change-Id: I184b445c43aa44cb000bfacc1cad18d2de265c48
---
M pywikibot/__init__.py
1 file changed, 7 insertions(+), 1 deletion(-)
Approvals:
Zhuyifei1999: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/__init__.py b/pywikibot/__init__.py
index 9c9632d..6fe82cc 100644
--- a/pywikibot/__init__.py
+++ b/pywikibot/__init__.py
@@ -1372,7 +1372,9 @@
'{lightblue}Waiting for {num} pages to be put. '
'Estimated time remaining: {sec}{default}', num=num, sec=sec))
- while _putthread.is_alive() and page_put_queue.qsize() > 0:
+ while (_putthread.is_alive()
+ and (page_put_queue.qsize() > 0
+ or page_put_queue_busy.qsize() > 0)):
try:
_putthread.join(1)
except KeyboardInterrupt:
@@ -1398,10 +1400,12 @@
"""Daemon; take requests from the queue and execute them in
background."""
while True:
(request, args, kwargs) = page_put_queue.get()
+ page_put_queue_busy.put(None)
if request is None:
break
request(*args, **kwargs)
page_put_queue.task_done()
+ page_put_queue_busy.get()
def async_request(request, *args, **kwargs):
@@ -1420,6 +1424,8 @@
# queue to hold pending requests
page_put_queue = Queue(config.max_queue_size)
+# queue to signal that async_manager is working on a request. See T147178.
+page_put_queue_busy = Queue(config.max_queue_size)
# set up the background thread
_putthread = threading.Thread(target=async_manager)
# identification for debugging purposes
--
To view, visit
https://gerrit.wikimedia.org/r/375448
To unsubscribe, or for help writing mail filters, visit
https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I184b445c43aa44cb000bfacc1cad18d2de265c48
Gerrit-Change-Number: 375448
Gerrit-PatchSet: 4
Gerrit-Owner: Mpaa <mpaa.wiki(a)gmail.com>
Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com>
Gerrit-Reviewer: Dvorapa <dvorapa(a)seznam.cz>
Gerrit-Reviewer: Framawiki <framawiki(a)tools.wmflabs.org>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Magul <tomasz.magulski(a)gmail.com>
Gerrit-Reviewer: Matěj Suchánek <matejsuchanek97(a)gmail.com>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: Zhuyifei1999 <zhuyifei1999(a)gmail.com>
Gerrit-Reviewer: jenkins-bot (75)