jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/789587 )
Change subject: [IMPR] Increase minimum workers ......................................................................
[IMPR] Increase minimum workers
doubling the workers to have one free for each family is a good value
Change-Id: I5fa697c8959ed5c70e8a76cf684981ff53dac55f --- M scripts/maintenance/preload_sites.py 1 file changed, 7 insertions(+), 4 deletions(-)
Approvals: Mpaa: Looks good to me, approved jenkins-bot: Verified
diff --git a/scripts/maintenance/preload_sites.py b/scripts/maintenance/preload_sites.py index 3c0afcd..5e0743b 100755 --- a/scripts/maintenance/preload_sites.py +++ b/scripts/maintenance/preload_sites.py @@ -73,15 +73,18 @@
.. versionchanged:: 7.3 Default of worker is calculated like for Python 3.8 but preserves - at least one worker more than families_list elements to ensure a - worker can be added in :func:`preload_family`. + at least one worker for each element in families_list for better + performance. """ start = datetime.now() if worker is None: # Python 3.8 default worker = min(32, (os.cpu_count() or 1) + 4) - # allow to add futures in preload_family - worker = max(len(families) + 1, worker) + # to allow adding futures in preload_family the workers must be one + # more than families are handled + worker = max(len(families) * 2, worker) + pywikibot.output('Using {} workers to process {} families' + .format(worker, len(families))) with ThreadPoolExecutor(worker) as executor: futures = {executor.submit(preload_family, family, executor) for family in families}