What is with this? Why are the XML dumps (the primary product of the projects: re-usable content) the absolute effing lowest possible effing priority? Why? I just finished (I thought) putting together some new software to update iwikis on the wiktionaries. It is set up to read the "langlinks" and "all-titles" part of the dumps. Just as I do that, the dumps fall down. Again. And no-one cares one whit; not even a reply here. (The bug was replied to after 4 days, and *might* be fixed presently, after 9 days?) My course of action now is to write new code to use thousands of API calls to get the information, albeit as efficiently as I can. When I do that, the chance that it will ever go back to using the dumps is a very close approximation to zero. After all, it will work somewhat better that way. Other people, *many*, *many*, other people are being *forced* to do the same, to maintain their apps and functions based on the WMF data. And there is no chance in hell they will go back to the dump "service" either. Brion, Tim, et al: you are worried about overall server load? Get the dumps working. This morning. And make it crystal clear that they will not break, and you will be checking them n times a day and they can be utterly, totally, absolutely relied upon. It's like that. People will use what *works*. Want people to use the dumps? Make them WORK. Want everyone to just dynamically crawl the live DB, with whatever screwy lousy inefficiency? FIne, just continue as you are, where that is all that can be relied upon! Look at the other threads: people asking if they can crawl the English WP at one per second, or maybe what? Is that what you want? That is what you are telling people to do, when the dump "service" says "2009-02-12 06:52:16 pswiki: Dump in progress" at the top on the 22nd of February. FYI for all others: if you want content dumps of the English Wiktionary, they are available in the usual XML format at http://devtionary.info/w/dump/xmlu/ at ~ 09;00 UTC. Every day. With my best regards, Robert On Tue, Feb 17, 2009 at 7:35 PM, Russell Blau <russblau(a)hotmail.com> wrote:

Robert Ullmann wrote: They are better than rants :) I don't think they move backup copies off to secure storage. They have the db replicated and the backup discs would be copies of that same dumps. (Some sysadmin to confirm?) The problem is that WMF can't provide that raw unfiltered information. Perhaps you could donate a box on the condition that it could only be used for dump processing, but giving out unfiltered data would be too risky.

The reason these dumps are not rewritten more efficiently is that this job was handed to me (at my request) and I have not been able to get to it, even though it is the first thing on my list for development work. So, if there are going to be rants, they can be directed at me, not at the whole team. The work was started already by a volunteer. As I am the blocking factor, someone else should probably take it on and get it done, though it will make me sad. Brion discussed this with me about a week and a half ago and I still wanted to keep it then but it doesn't make sense. The in-office needs that I am also responsible for take virtually all of my time. Perhaps they shouldn't, but that is how it has worked out. So, I am very sorry for having needlessly held things up. (I also have aa crawler that requests pages changed since the latest xml dump, so that projects I am on can keep a current xml file; we've been running that way for at least a year.) Ariel Στις 23-02-2009, ημέρα Δευ, και ώρα 00:37 +0100, ο/η Gerard Meijssen έγραψε:

yep, http://svn.wikimedia.org/viewvc/mediawiki/trunk/backup/ +) 2009/2/23 Alex <mrzmanwiki(a)gmail.com>om>: -- Nicolas Dumazet — NicDumZ [ nɪk.d̪ymz ]

"Lars Aronsson" <lars(a)aronsson.se> wrote in message news:Pine.LNX.4.64.0902231202140.1043@localhost.localdomain... ... I have to second this. I tried to report this outage several times last week - on IRC, on this mailing list, and on Bugzilla. All reports -- NOT COMPLAINTS, JUST REPORTS -- were met with absolute silence. I fully understand that time and resources are limited, and not everything can be fixed immediately, but at least some acknowledgement of the reports would be appreciated. It is extremely disheartening to members of the user community of what is supposed to be a collaborative project when attempts to contribute by reporting a service outage are ignored. Russ

Thanks for the update Russell! On Feb 23, 2009, at 10:04 AM, Russell Blau wrote:

On Tue, Feb 24, 2009 at 1:07 PM, Robert Ullmann <rlullmann(a)gmail.com> wrote: Constructive suggestions for improvement are far more welcome than complaints and outrage. If you have no suggestions for improvement, it is perhaps more prudent to express concern that dumps are not working and to wait for a response. This is admittedly less fun than piecing together information and "lining up" those responsible for something not being operational. -- Andrew Garrett

Let me ask a separate question (Ariel may be interested in this): What if we took the regular permanent media backups, and WMF filtered them in house just to remove the classified stuff (;-), and then put them somewhere where others could convert them to the desired format(s)? (Build all-history files, whatever.) What is the standard backup procedure? (I ask as I haven't seen any description or reference to it ... :-) Robert

I'm not sure where (or if it has changed), but they are running now .... (:-) To Ariel Glenn: On getting them to work better in the future, this is what I would suggest: First, note that everything except the "all history" dumps presents no problem. It isn't perfect, but it is workable. The biggest "all pages current" dump is enwiki, which takes about a day and a half, and the compressed output file (bz2) still fits neatly on a DVD. As to the history files, these are the problem; each contains all of the preceding history and they just grow and grow. They must be partitioned somehow. Suggestions have been made concerning alphabetical partitions (very traditional for encyclopaedias ;-); you yourself suggested page id. I suggest the history be partitioned into "blocks" by *revision ID* Like this: revision IDs (0)-999,999 go in "block 0", 1M to 2M-1 in "block 1", and so on. The English Wiktionary at the moment would have 7 blocks; the English Wikipedia would have 273. The dumps would continue as now up to "all pages current", including the split-stub dump for the history (very important, as it provides the "snapshot" of the DB state). But then when it gets to history, it re-builds the last block done (possibly completing it), and then writes 0-n new ones as needed. Note that (to pick a random number) "block 71" of the enwiki defined this way *has not changed* in a long time; only the current block(s) need to be (re-)written. The history stays the same. (Of course?!) If someone somewhere needs a copy of the wiki with all history as of a given date, they can start with the split-stub for that date and read in all the required blocks. But that isn't your problem any more. (;-) They can do that with their disk and servers. It would probably be best to still sort by page-id order within each block, as they will compress much better that way. One reason to rebuild the last block (or two) is to filter out deleted and oversighted revisions. Deleted and oversighted revisions older than some specific time (a small number of weeks) would remain. But note that that is true *anyway*, as someone can always look at a 3-month old dump under any method. With my best regards, Robert

But server space saved by compression would be would be compensated by the stability, and flexibility provided by this method. this would allow what ever server is controlling the dump process to designate and delegate parallel processes for the same dump. so block 1 could be on server 1 and block 2 could be on server 3. that would give the flexibility to use as many servers as are available for this task more efficiently. if block 200 of en.wp breaks for some reason you dont have to rebuild the previous 199 blocks you can just delegate a server to rebuild that single block. that would allow the dump process to be a little more crash friendly (even though I know we dont want to admit crashes happen :) ) this also enables the dump time in future dumps to be cut drasticlly. Id recommend either 10m or 10% of the database which ever is larger for new dumps to screen out a majority of the deletions. what are your thoughts on this process brion (and the rest of the tech team)? Betacommand On Wed, Feb 25, 2009 at 9:00 AM, Thomas Dalton <thomas.dalton(a)gmail.com>wrote;wrote:

2009/2/25 John Doe <phoenixoverride(a)gmail.com>om>: Another idea: If $revision is deleted/oversighted/whateverhowmadeinvisible, then find out the block ID for the dump so that only this specific block needs to be re-created in next dump run. Or, better: do not recreate the dump block, but only remove the offending revision(s) from it. Shoulda save a lot of dump preparation time, IMO. Marco -- VMSoft GbR Nabburger Str. 15 81737 München Geschätsführer: Marco Schuster, Volker Hemmert http://vmsoft-gbr.de

On Thu, Feb 26, 2009 at 5:08 AM, John Doe <phoenixoverride(a)gmail.com> wrote: Not nearly -- we're talking about a 100-fold decrease in compression ratio if we don't compress revisions of the same page adjacent to one another. -- Andrew Garrett

Robert Ullmann wrote: Not only do you need to keep them in the same block. You also need to keep them inside the compression window. Unless you are going to reorder those 1M revisions to keep revisions to the same article together.

On Tue, Feb 24, 2009 at 5:09 PM, Robert Ullmann &lt;rlullmann(a)gmail.com&gt; wrote: <snip> <snip> Though there are arguments in favor of this, I think they are outweighed by the fact that one would need to go through every block in order to reconstruct the history of even a single page. In my opinion partitioning on page id is a much better idea since it would keep each page's history in a single place. -Robert Rohde

2009/2/24 Robert Ullmann &lt;rlullmann(a)gmail.com&gt;om>: Is there anyone within minutes of the servers at all times? Aren't they at a remote data centre?

2009/2/24 Aryeh Gregor &lt;Simetrical+wikilist(a)gmail.com&gt;om>: He's based somewhere near the data centre, but I'm not sure he's actually there unless there is something which needs his attention. He's certainly not there 24/7 (regrettably, WMF is still using human sysadmins...).

afaik there are "hands" in amsterdam that can be called upon to do stuff as necessary in the centre like any other hosting customer, but the need is not quite of the same level as tampa due to size, servers there etc. seoul no longer operates so this is not an issue. regards mark On Tue, Feb 24, 2009 at 2:55 PM, Gerard Meijssen &lt;gerard.meijssen(a)gmail.com&gt;wrote;wrote:

On Mon, Feb 23, 2009 at 11:08 AM, Alex &lt;mrzmanwiki(a)gmail.com&gt; wrote: In a "very broad sense" (and not directly connected to main problems), I wrote a compressor [1] that converts full-text history dumps into an "edit syntax" that provides ~95% compression on the larger dumps while keeping it in a plain text format that could still be searched and processed without needing a full decompression. That's one of several ways to modify the way dump process operates in order to make the output easier to work with (if it takes ~2 TB to expand enwiki's full history, then that is not practical for most users even if we solve the problem of generating it). It is not necessarily true that my specific technology is the right answer, but various changes in formatting to aid distribution, generation, and use are one of the areas that ought to be considered when reimplementing the dump process. The largest gains are almost certainly going to be in parallelization though. A single monolithic dumper is impractical for enwiki. -Robert Rohde [1] http://svn.wikimedia.org/viewvc/mediawiki/trunk/tools/editsyntax/

Ariel, Thank you for giving some insight into what has been going on behind the scenes. I have a few questions that will hopefully get some answers to those of us eager to help out in any way we can. What are the planned code changes to speed the process up? Can we help this volunteer with the coding or architectural decisions? How much time do they have to dedicate to it? Some visibility into the fix and timeline would benefit a lot of us. It would also help us know how we can help out! Thanks again for shedding some light on the issue. On Feb 22, 2009, at 8:12 PM, Ariel T. Glenn wrote:

Στις 23-02-2009, ημέρα Δευ, και ώρα 19:02 +0000, ο/η Thomas Dalton έγραψε: I asked for it, and that's why it was assigned to me. I should have recognized much sooner that I could not actually get it done and should have brought this to Brion's attention instead of continuing to hang on to it after he brought it to my attention. Ariel

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Robert Ullmann: wrong list. the Foundation needs to allocate the resources to fix dumps. it hasn't done so, therefore dumps are still broken. perhaps you might ask the Foundation why dumps have such a low priority. - river.

On 2/23/09 3:08 AM, Marco Schuster wrote: Define alphabet -- how should Chinese and Japanese texts be broken up? We're much more likely to break them up simply by page ID. Not currently. By third parties, with AFAIK very little usage. -- brion