Tim Starling: Is such a field really needed? The software could assemble an URI from existing information like http://$server/$project/$rev_id. This would also be more flexible because if data needs to be transferred to another server there is no need to update all rows in the revision table. But I don't want to give unwanted advice here. The background of my question is that I have written a Perl program that compresses page histories much better than the currently used algorithm. And now I want to write PHP code so that MediaWiki can access the data. But HistoryBlobStubs make this more complicated. This is how my method works: All revision texts are split into sections (the delimiter is "\n=="). Unchanged sections are stored only once. Sections are sorted by their headings. Then everything is compressed with deflate(). This has several advantages: * Less memory is used because many sections don't change from revision to revision. This is especially true for discussion pages. * Compression and decompression is fast because deflate() is used and it needs to (de)compress less data. * If the pages are larger than 32kB, compression is much better than can be achieved by simply concatenating revisions and compressing them with deflate() and even bzip2. On average, my method compresses about five times better than the currently used one. I have listed some values on http://meta.wikipedia.org/wiki/User:El/History_compression Now that I've read that the revision texts are about to be relocated to external storage servers I wonder if it would be worth to write the PHP classes at all. What software is going to run on the storage servers? Is my method still competitive there? -- 5 GB Mailbox, 50 FreeSMS http://www.gmx.net/de/go/promail +++ GMX - die erste Adresse f�r Mail, Message, More +++

On 27/05/05, Brion Vibber <brion(a)pobox.com> wrote: Updating $serverurl or similar in a PHP file is easier than using str_replace or similar for every entry of your revision table to change the URL from the old to the new one.

On Fri, May 27, 2005 at 11:38:27PM +0100, Tomer Chachamu wrote: The current implementation uses logical names within the database and a logical name -> physical server mapping in LocalSettings.php. The URL for an external revision is DB://cluster1/765765 In LocalSettings.php, the name cluster1 is mapped to a farm of three mysql servers: $wgExternalServers = array( 'cluster1' => array( array( 'host'=> 'srv30', 'load' =>1)+$templateServer , array( 'host'=> 'srv28', 'load' =>1)+$templateServer , array( 'host'=> 'srv29', 'load' =>1)+$templateServer , ) ); This provides both redundancy and flexibility. Regards, JeLuF

Timwi: Yes, if both revision texts reside in the same history blob. Up to 20 consecutive revisions are stored in one blob. No. I doubt that this would improve the compression much, because texts aren't copied that often. The complete history blob must be decompressed of course. But no previous revisions need to be reconstructed. At the beginning of the uncompressed history blob there is a section index for each revision followed by a list of (position, length)-pairs for each section. So when a revision text is to be extracted, this is what happens: * uncompress history blob * look up section list for the requested revision * loop up section offsets and lengths * concatenate sections This is an example header (first 20 revisions of the german article "Stern"): 00000020 00000025 00000142 00000260 00000001 # 20 revisions, 25 different sections 0 # first revision has no heading: only one section 1 2 3 4 5 6 4 5 6 # conversion script: nothing changed 7 5 8 7 9 8 10 9 8 7 9 8 7 11 8 12 11 8 12 13 8 12 14 8 15 14 8 16 14 8 17 14 8 18 14 8 19 14 8 19 20 8 21 22 23 21 24 23 0 2579 # offset and length of the first section 2579 1176 ... -- Weitersagen: GMX DSL-Flatrates mit Tempo-Garantie! Ab 4,99 Euro/Monat: http://www.gmx.net/de/go/dsl