jayvdb created this task. jayvdb claimed this task. jayvdb added a subscriber: jayvdb. jayvdb added a project: pywikibot-core. jayvdb changed Security from none to none.

TASK DESCRIPTION When set_maximum_items is set to -1, as is done to fetch revisions, or in the default None state, the argument given to set_query_increment is ignored. This means it does not always act like 'step' which is commonly available to the site API , and APISite._generator 'step' argument is put into a set_query_increment call.

This effects the responsiveness of the generators, and other aspects like network & memory utilisation. If the caller wants to process a lot of properties for a large list of items, but prefers to do this in small batches, QueryGenerator will attempt to process all items (it will actually send more items than the API limit for titles, pageids, etc), and will iterate over them all.

QueryGenerator, or a subclass, should send batches of the requested query increment only, and continue sending batches until the requested set of items is processed.

TASK DETAIL https://phabricator.wikimedia.org/T76141

REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.

To: jayvdb Cc: pywikipedia-bugs, jayvdb