Lars Aronsson wrote: No. AFAIK it is done as follows: Precondition: The last full dump (if not present, treat as empty). 1- Take an snapshot of the wiki status (page table?) and create stub-meta-history 2- Read stub-meta-history and fill the page content with the last dump page contents. If a page content is not on previous dump, get it from the external storage in a blocking way. Result: A bzipped2 full history dump. The bzip2 dump is then uncompressed and 7zipped. If there's an error on a call to the external storage, the process can't be resumed and the dump fails. I had been recently thinking on it and think it could be done as this: Precondition: The last full dump (if not present, treat as empty) and its greatest revid. 1a- Take an snapshot of the wiki status (page table?) and create stub-meta-history 1b- While reading the revisions, if revid is greater than the lastdumpgreaterrevid (LDGR), add it to N files (a file per M revisions). 2-Run N processes grabbing these page contents. Store them on a new-format dump (the external storage equivalent), one per revid list file. If one fails, just rerun it. 3- Read stub-meta-history and fill the page content with the last dump page contents. If a page text is not on previous dump, grab from the list file if revid > LDGR else, get it from the external storage saving it on a different file. Revisions not present on last dump nor incremental dumps will occur on restored pages, and still be able to block it, but being much less, it's much more unlikely that they fail. 4-Save the new dump LDGR with the new bzipped dump. Making available the M+1 incremental dumps, using the smaller meta-stubs-history, last dump can be recreated using the previous one (=less download size). Wikimedia would still provide the full dumps, but you would only be need ed the first time. Comments? _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikitech-l

Bleh. Someone pulling increments couldn't build a point in time snapshot, they would need to always pull the full. And we want people using point in time versions of the site not mangled mixes.

Also, I expect that once 7zed the incremets will not be too much smaller than the full, especially if partitoned by revid.

On 10/20/07, Platonides <Platonides(a)gmail.com> wrote: Okay, you didn't mention that.... but please no: I have had a hard enough time explaining to people that the separate SQL dumps aren't consistent with the history dumps. I don't want to end up in a situation where the only way to get a sane copy of the site is stitching together dozens of files on the recipients side.... people will do it wrong, or just skip building a point in time version at all.. and make a big mess. I'd rather go back to having separate metadata and text dumps than end up with people needing to combine an old full dump, N large incremental files, and a new stub dump through a bunch of complex manipulation in order to arrive at a consistent copy of the site. If we wanted to do that on the back end.. fine.

Partition by revid doesn't necessarily mean one rev per file... and thats certainly not what I thought you were suggesting. You will screw compression if you partition by revid (i.e. in groups of revs, failing to keep all revs of a single article in one place). If you don't want to take my word for it try it yourself.