Bugs item #2619054, was opened at 2009-02-20 03:04
Message generated for change (Comment added) made by russblau
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603138&aid=261905…
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: rewrite
Group: None
Status: Pending
Resolution: Fixed
Priority: 5
Private: No
Submitted By: NicDumZ — Nicolas Dumazet (nicdumz)
Assigned to: Russell Blau (russblau)
Summary: clarify between limit, number, batch and step parameters
Initial Comment:
I had a strange behavior of replace.py -weblink: that I couldn't quite diagnose: some
pages were not treated.
First of all, those detailed logs are a great gift. They are a bit messy to understand at
first, but thanks to those I found the bug and fixed it in r6386 (
http://svn.wikimedia.org/viewvc/pywikipedia?view=rev&revision=6386 ).
I believe that this parameter confusion is a very bad habit we have from the old
framework. (the only reason there we have those bugs is because we merged pagegenerators
from trunk.) We need to agree on common parameters for generators that have a global
meaning, and stick to it.
I personally think that -limit might be a bit confusing (is it an api limit, a limit
enforced by the local application on a huge fetched set, etc ?), while -number appears a
bit more clear. But it's a personal opinion =)
What about -number for "number of items to retrieve", and -step, or -maxstep for
the maximum number of items to retrieve at once ?
Actually, I don't mind about the names; we just need to agree on something meaningful
enough, and document them in the file headings.
On a sidenote, replace.py -fix:yu-tld -weblink:*.yu is actually running on fr.wp. No
issues sighted. =)
----------------------------------------------------------------------
Comment By: Russell Blau (russblau)
Date:
2010-01-06 13:59
Message:
This was fixed a while back but I neglected to close the bug; please reopen
if any continuing problems exist.
----------------------------------------------------------------------
Comment By: NicDumZ — Nicolas Dumazet (nicdumz)
Date: 2009-02-21 23:50
Message:
Well I think that one of the first steps here is to consider what is
currently done in the old pagegenerators =)
Here's a small summary of the "limits" enforced by our old
pagegenerators.
The overall internal naming consistency factor is quite low for now, not
to mention the surprising facts I found :s
I've considered for each generator, the pagegenerators function, and its
Site/Page/Image/Category counterpart: unless noted, both function parameter
namings are consistent.
* shortpages, new(pages|images), unusedfiles, withoutinterwiki,
uncategorized(images|categories|pages), unwatchedpages, ancientpages,
deadendpages, longpages, shortpages, search
They use "number" (meant as "batch"/"max") + boolean
"repeat". Overall,
you can get either "number" items, or all.
* random(page|redirect) are good examples of inconsistencies:
they use number (batch/max) + repeat, but since Special:Random gives only
one page at a time, the actual "batch" parameter is always 1. (behavior is
"for _ in range(number), fetch one page")
And if repeat=True ... those functions never stop, if I'm right.
irrrk !!
* filelinks, imagelinks, interwiki
they scrap the article wikipage, and yield everything in one step from the
wikitext
* categorymembers, subcategories
they scrap category pages. No parameter is available, since the UI doesn't
let us customize the number of displayed links. Follows the (next) links on
the category page. Stops when all the items have been retrieved.
* allpages, prefixindex, getReferences
no function parameters. They use config.special_page_limit as "batch/max",
and all items are retrieved through repeated queries.
if special_page_limit > 999, getReferences sets it back to 999. (?!)
* linksearch
pagegenerators has a "step=500" parameter, the corresponding Site function
uses "limit=500". Meant as "batch/batch": all the links are retrieved
through repeated queries
* usercontribs
number=250, meant as "batch/max". All the contribs are retrieved through
repeated queries. if number>500, sets it back to 500
It seems that the most common used combination is number+repeat. But I
really don't think that it is the way to go, since you cannot accurately
describe the total number of items you want to retrieve: either number,
either all items...
I think a "batch" + "total" integer parameters could be more useful
here
(namings are illustrative)
On the other hand, users should be able to say "I want to retrieve all the
items": looking into the code, I see that a "-1" convention is used now.
If
I understand things correctly, it is used in a "batch" context: if we call
set_maximum_items(-1), in most of the cases, the API uses its default
xxlimit number. We could use such a convention for our "total" parameter
too. Be it -1, or None, whatever, but I think that with such a policy, we
should cover all the use cases.
Given what I found, I really don't think that backwards compatibility
should be a priority here. I would rather introduce a breaking change in
namings, so that people don't expect the new limits to work "as in the old
framework"... because in the old framework, limit behaviors were not even
internally consistent...
----------------------------------------------------------------------
Comment By: Russell Blau (russblau)
Date: 2009-02-20 10:00
Message:
A good point. A query can have two different types of limits: the limit on
the number of pages/links/whatever retrieved from the API in a single
request (defaults to "max"), and the limit on the total number of items to
be retrieved from a repeated query. We should do this in a way that is (a)
internally consistent among all generators, and (b) as much as possible,
backwards-compatible with the old pagegenerators module (but this is
secondary to getting something that works).
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603138&aid=261905…