Hi Roan,

thanks for the answers!

> You then introduce new parameters: startid, endid, start, end (for start/end
> of revid, or start/end of last touched), and amend the query:
> if (isset ($params['start'])) {
> $this->addWhere('page_touched>=' . $params['start']);
> }
>
> Finally you need something like:
> $this->addOption('ORDER BY', 'page_touched');
> and
> $this->setContinueEnumParameter('start',
> $this->keyToTitle($row->page_latest));

Since there's no index on page_latest, sorting and paging on it the
way you do is inefficient. Especially the ORDER BY page_latest part
causes a filesort of the entire page table, which has over 10 million
entries on English Wikipedia.

I guess it would be the same if one sorted on revision id (rather than page_latest)?

Is there a proposal one could forward to make this more efficient, by somehow also indexing on revision id?

> that people can have a look?

This'll probably work (albeit breaking a few things such as apfrom, as
you mentioned), but due to the inefficient queries involved, it won't
make it into the MediaWiki core.

Again, is there something that could be done to make it more efficient?

Or perhaps one could put some less efficient code in, but with a switch to disable it on large wikis?

Just to give a little background, why I think this is important: Mediawiki is an important platform for Open Educational Resources, and when considering scenarios in developing countries, bandwidth is expensive, and so mirroring is important. Of course once you mirror, you want to start up to date, so being able to get updates since a revision or a date is important.

Would really like to hear your ideas on what the best way forward is!

Thanks again,

Bjoern