Hi Roan,
thanks for the answers!
You then introduce new parameters: startid, endid, start, end (for
start/end
of revid, or start/end of last touched), and amend the query: if (isset ($params['start'])) { $this->addWhere('page_touched>=' . $params['start']); }
Finally you need something like: $this->addOption('ORDER BY', 'page_touched'); and $this->setContinueEnumParameter('start', $this->keyToTitle($row->page_latest));
Since there's no index on page_latest, sorting and paging on it the way you do is inefficient. Especially the ORDER BY page_latest part causes a filesort of the entire page table, which has over 10 million entries on English Wikipedia.
I guess it would be the same if one sorted on revision id (rather than page_latest)?
Is there a proposal one could forward to make this more efficient, by somehow also indexing on revision id?
that people can have a look? This'll probably work (albeit breaking a few things such as apfrom, as you mentioned), but due to the inefficient queries involved, it won't make it into the MediaWiki core.
Again, is there something that could be done to make it more efficient?
Or perhaps one could put some less efficient code in, but with a switch to disable it on large wikis?
Just to give a little background, why I think this is important: Mediawiki is an important platform for Open Educational Resources, and when considering scenarios in developing countries, bandwidth is expensive, and so mirroring is important. Of course once you mirror, you want to start up to date, so being able to get updates since a revision or a date is important.
Would really like to hear your ideas on what the best way forward is!
Thanks again, Bjoern