English -> English dump
Because myself and others have been frustrated by the lack of good stats on the number of active editors on the English Wikipedia, I have compiled some stats on the editing frequency on enwiki:
No worries: in only 176 days from now the English dump will be ready and I can run wikistats scripts on it. It just started 52 days ago, so let us be patient for a while ;)
http://www.infodisiac.com/cgi-bin/WikimediaDownload.pl
Nice stats though!
Erik Zachte
On Sun, Nov 30, 2008 at 8:20 AM, Erik Zachte erikzachte@infodisiac.com wrote:
English -> English dump
Because myself and others have been frustrated by the lack of good stats on the number of active editors on the English Wikipedia, I have compiled some stats on the editing frequency on enwiki:
No worries: in only 176 days from now the English dump will be ready and I can run wikistats scripts on it. It just started 52 days ago, so let us be patient for a while ;)
Is there any reason at all to believe that it is more likely to finish this time than all the previous attempts during the last two years?
I have virtually zero faith in a script that takes 230 days and where any error wipes out all progress.
-Robert Rohde
On 30 Nov 2008, at 20:11, Robert Rohde wrote:
On Sun, Nov 30, 2008 at 8:20 AM, Erik Zachte erikzachte@infodisiac.com wrote:
English -> English dump
Because myself and others have been frustrated by the lack of good stats on the number of active editors on the English Wikipedia, I have compiled some stats on the editing frequency on enwiki:
No worries: in only 176 days from now the English dump will be ready and I can run wikistats scripts on it. It just started 52 days ago, so let us be patient for a while ;)
Is there any reason at all to believe that it is more likely to finish this time than all the previous attempts during the last two years?
I have virtually zero faith in a script that takes 230 days and where any error wipes out all progress.
-Robert Rohde
Hold on...what? There is no recent dump of the English Wikipedia, and there hasn't been for the last 2 years?
Please tell me I'm misunderstanding things here.
Mike
On Sun, Nov 30, 2008 at 3:35 PM, Michael Peel email@mikepeel.net wrote:
On 30 Nov 2008, at 20:11, Robert Rohde wrote:
On Sun, Nov 30, 2008 at 8:20 AM, Erik Zachte erikzachte@infodisiac.com wrote:
English -> English dump
Because myself and others have been frustrated by the lack of good stats on the number of active editors on the English Wikipedia, I have compiled some stats on the editing frequency on enwiki:
No worries: in only 176 days from now the English dump will be ready and I can run wikistats scripts on it. It just started 52 days ago, so let us be patient for a while ;)
Is there any reason at all to believe that it is more likely to finish this time than all the previous attempts during the last two years?
I have virtually zero faith in a script that takes 230 days and where any error wipes out all progress.
-Robert Rohde
Hold on...what? There is no recent dump of the English Wikipedia, and there hasn't been for the last 2 years?
Please tell me I'm misunderstanding things here.
Mike
foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
(cc'd to wikitech-l) I saw this the other day as well and found it odd. While enwiki dumps do take the longest, this does seem like an _incredibly_ long time for "All pages with complete page edit history (.bz2)" to finish (May 2009).
-Chad
I saw this the other day as well and found it odd. While enwiki dumps do take the longest, this does seem like an _incredibly_ long time for "All pages with complete page edit history (.bz2)" to finish (May 2009).
Do you know how many pages enwiki has and how much edit history they each have? It's a lot!
I think the dumps work by starting with the last successful dump and just adding in anything that's changed, but because there haven't been any successful dumps of the whole of enwiki in a long time, it basically has to start from scratch, which is going to take a long time (and means it probably won't succeed - ie. we have a catch-22). It seems to me that (if my understanding of the problem is correct), the answer is to devote a more powerful computer to the dump for just this one so that we can get things moving again - I'm sure if we asked around someone could lend us a really powerful computer for a few weeks to do the dump on.
On Sun, Nov 30, 2008 at 12:58 PM, Thomas Dalton thomas.dalton@gmail.com wrote:
I saw this the other day as well and found it odd. While enwiki dumps do take the longest, this does seem like an _incredibly_ long time for "All pages with complete page edit history (.bz2)" to finish (May 2009).
Do you know how many pages enwiki has and how much edit history they each have? It's a lot!
I think the dumps work by starting with the last successful dump and just adding in anything that's changed, but because there haven't been any successful dumps of the whole of enwiki in a long time, it basically has to start from scratch, which is going to take a long time (and means it probably won't succeed - ie. we have a catch-22). It seems to me that (if my understanding of the problem is correct), the answer is to devote a more powerful computer to the dump for just this one so that we can get things moving again - I'm sure if we asked around someone could lend us a really powerful computer for a few weeks to do the dump on.
No, dumps are total, not incremental.
It is really more than throwing a big computer at it. The dumping process ought to be redesigned to be more fault tolerant and faster. It is ridiculous to have a process that is expected to take months and yet have no method of saving one's progress as it goes and restarting it in case of trouble.
-Robert Rohde
No, dumps are total, not incremental.
The dumps are, yes, but what about the creation of the dumps? My understanding was that they are created by taking an old dump and adding to it.
On Sun, Nov 30, 2008 at 2:39 PM, Thomas Dalton thomas.dalton@gmail.com wrote:
No, dumps are total, not incremental.
The dumps are, yes, but what about the creation of the dumps? My understanding was that they are created by taking an old dump and adding to it.
No. Each dump process starts fresh from the database, they don't add to old dumps. (Or at least they certainly didn't the last time I looked at the dumping scripts, and I have no reason to suspect that has changed.)
-Robert
2008/11/30 Michael Peel email@mikepeel.net:
Hold on...what? There is no recent dump of the English Wikipedia, and there hasn't been for the last 2 years?
Please tell me I'm misunderstanding things here.
Mike
Not to years but yes the English Wikipedia dumps very rarely work. De.wikipedia is starting to suffer the same issues and image database dumps don't happen.
On Sun, Nov 30, 2008 at 12:56 PM, geni geniice@gmail.com wrote:
2008/11/30 Michael Peel email@mikepeel.net:
Hold on...what? There is no recent dump of the English Wikipedia, and there hasn't been for the last 2 years?
Please tell me I'm misunderstanding things here.
Mike
Not to years but yes the English Wikipedia dumps very rarely work. De.wikipedia is starting to suffer the same issues and image database dumps don't happen.
No, I think it really has been two years (September 2006 to be precise). I'm pretty sure there have been no complete dumps this calendar year. I believe there was one (or two?) full dump process that claimed to run to completion in 2007 but it was later found to have been truncated (i.e. it didn't really dump all of enwiki, only a portion of it).
If I am wrong and there really is a more recent complete history dump of enwiki floating around somewhere, then I'd love to hear about it, but I don't believe that is the case.
-Robert Rohde
Not to years but yes the English Wikipedia dumps very rarely work. De.wikipedia is starting to suffer the same issues and image database dumps don't happen.
No, I think it really has been two years (September 2006 to be precise). I'm pretty sure there have been no complete dumps this calendar year. I believe there was one (or two?) full dump process that claimed to run to completion in 2007 but it was later found to have been truncated (i.e. it didn't really dump all of enwiki, only a portion of it).
If I am wrong and there really is a more recent complete history dump of enwiki floating around somewhere, then I'd love to hear about it, but I don't believe that is the case.
That sounds about right to me. I think the confusion may come from there being lots of different dumps - the smaller dumps of enwiki do succeed (occasionally, at least!), it's the full dump of every revision of every page that fails routinely.
Thomas Dalton wrote:
Not to years but yes the English Wikipedia dumps very rarely work. De.wikipedia is starting to suffer the same issues and image database dumps don't happen.
No, I think it really has been two years (September 2006 to be precise). I'm pretty sure there have been no complete dumps this calendar year. I believe there was one (or two?) full dump process that claimed to run to completion in 2007 but it was later found to have been truncated (i.e. it didn't really dump all of enwiki, only a portion of it).
If I am wrong and there really is a more recent complete history dump of enwiki floating around somewhere, then I'd love to hear about it, but I don't believe that is the case.
That sounds about right to me. I think the confusion may come from there being lots of different dumps - the smaller dumps of enwiki do succeed (occasionally, at least!), it's the full dump of every revision of every page that fails routinely.
foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Is the data replicated anywhere outside the Tampa data centre (such as in Amsterdam or Seoul)? If not, just one fire, flood or hurricane could destroy the entire en: Wikipedia.
-- Neil
2008/12/1 Neil Harris usenet@tonal.clara.co.uk:
Is the data replicated anywhere outside the Tampa data centre (such as in Amsterdam or Seoul)? If not, just one fire, flood or hurricane could destroy the entire en: Wikipedia.
En probably isn't that bad. We have lost commons images already.
On Sun, Nov 30, 2008 at 6:16 PM, Neil Harris usenet@tonal.clara.co.uk wrote:
Is the data replicated anywhere outside the Tampa data centre (such as in Amsterdam or Seoul)? If not, just one fire, flood or hurricane could destroy the entire en: Wikipedia.
There are database mirrors of every wiki, including en, as part of the toolserver cluster in Amsterdam.
-Robert Rohde
It is good to hear that en: is replicated. I assume this also applies to commons?
Still, it might be a good idea to think about a redesign of the dump process.
On Mon, Dec 1, 2008 at 4:09 AM, Robert Rohde rarohde@gmail.com wrote:
On Sun, Nov 30, 2008 at 6:16 PM, Neil Harris usenet@tonal.clara.co.uk wrote:
Is the data replicated anywhere outside the Tampa data centre (such as in Amsterdam or Seoul)? If not, just one fire, flood or hurricane could destroy the entire en: Wikipedia.
There are database mirrors of every wiki, including en, as part of the toolserver cluster in Amsterdam.
-Robert Rohde
foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
On Sun, Nov 30, 2008 at 8:04 PM, teun spaans teun.spaans@gmail.com wrote:
It is good to hear that en: is replicated. I assume this also applies to commons?
The database yes. For images there is as-needed caching in Amsterdam and Korea, but my understanding is that the only complete copies of the file store all live in Florida.
On a related point, replicated mirrors deal well enough with acts of God (e.g. if a meteor hits the data center in Florida), but because everything is live mirrored it doesn't deal well if the server is accidentally instructed to delete something. That erroneous instruction would be carried to all the mirrors as well, and you might really lose something before the links could be severed. We really need to have periodic stored backups in addition to the replicated mirrors.
-Robert Rohde
On Monday 01 December 2008 04:09:11 Robert Rohde wrote:
On Sun, Nov 30, 2008 at 6:16 PM, Neil Harris usenet@tonal.clara.co.uk
wrote:
Is the data replicated anywhere outside the Tampa data centre (such as in Amsterdam or Seoul)? If not, just one fire, flood or hurricane could destroy the entire en: Wikipedia.
There are database mirrors of every wiki, including en, as part of the toolserver cluster in Amsterdam.
Unfortunately, enwiki mirror doesn't include article text :(
On Sun, Nov 30, 2008 at 10:29 PM, Nikola Smolenski smolensk@eunet.yu wrote:
On Monday 01 December 2008 04:09:11 Robert Rohde wrote:
On Sun, Nov 30, 2008 at 6:16 PM, Neil Harris usenet@tonal.clara.co.uk
wrote:
Is the data replicated anywhere outside the Tampa data centre (such as in Amsterdam or Seoul)? If not, just one fire, flood or hurricane could destroy the entire en: Wikipedia.
There are database mirrors of every wiki, including en, as part of the toolserver cluster in Amsterdam.
Unfortunately, enwiki mirror doesn't include article text :(
Ouch, I hadn't realized they gave up on text replication. Apparently quite a while ago too. (That will teach me for never bothering to learn to use the toolserver.)
So I guess we are back to the meteor impact destroys Wikipedia scenario.
-Robert Rohde
Robert Rohde wrote:
On Sun, Nov 30, 2008 at 10:29 PM, Nikola Smolenski smolensk@eunet.yu wrote:
On Monday 01 December 2008 04:09:11 Robert Rohde wrote:
On Sun, Nov 30, 2008 at 6:16 PM, Neil Harris usenet@tonal.clara.co.uk
wrote:
Is the data replicated anywhere outside the Tampa data centre (such as in Amsterdam or Seoul)? If not, just one fire, flood or hurricane could destroy the entire en: Wikipedia.
There are database mirrors of every wiki, including en, as part of the toolserver cluster in Amsterdam.
Unfortunately, enwiki mirror doesn't include article text :(
Ouch, I hadn't realized they gave up on text replication. Apparently quite a while ago too. (That will teach me for never bothering to learn to use the toolserver.)
So I guess we are back to the meteor impact destroys Wikipedia scenario.
-Robert Rohde
That's scary: are there any off-site backups of the full database, including the article text, made anywhere, other than the constantly failing dumps? Given that Wikipedia is the Wikimedia Foundation's principal asset, I would hope that fixing this single point of failure would be a priority for the Foundation.
-- Neil
On Mon, Dec 1, 2008 at 7:29 AM, Nikola Smolenski smolensk@eunet.yu wrote:
On Monday 01 December 2008 04:09:11 Robert Rohde wrote:
On Sun, Nov 30, 2008 at 6:16 PM, Neil Harris usenet@tonal.clara.co.uk
wrote:
Is the data replicated anywhere outside the Tampa data centre (such as in Amsterdam or Seoul)? If not, just one fire, flood or hurricane could destroy the entire en: Wikipedia.
There are database mirrors of every wiki, including en, as part of the toolserver cluster in Amsterdam.
Unfortunately, enwiki mirror doesn't include article text :(
Are you sure about that? Last time I checked the text databases were shared between all wikimedia project and thus replicated all at once or not at all.
Bryan
2008/12/1 Bryan Tong Minh bryan.tongminh@gmail.com:
On Mon, Dec 1, 2008 at 7:29 AM, Nikola Smolenski smolensk@eunet.yu wrote:
On Monday 01 December 2008 04:09:11 Robert Rohde wrote:
On Sun, Nov 30, 2008 at 6:16 PM, Neil Harris usenet@tonal.clara.co.uk
wrote:
Is the data replicated anywhere outside the Tampa data centre (such as in Amsterdam or Seoul)? If not, just one fire, flood or hurricane could destroy the entire en: Wikipedia.
There are database mirrors of every wiki, including en, as part of the toolserver cluster in Amsterdam.
Unfortunately, enwiki mirror doesn't include article text :(
Are you sure about that? Last time I checked the text databases were shared between all wikimedia project and thus replicated all at once or not at all.
Bryan
http://upload.wikimedia.org/wikipedia/commons/f/ff/Wikimedia-servers-2008-11... is a good picture of it. If I'm interpreting it correctly:
* There are the "database servers" running MySQL, which store metadata. These are split into clusters (en on 1; de + commons + possibly others on 2; others on 3). I believe these are all located in Florida. * These databases (i.e. only metadata) are replicated for the Toolserver machines located in Amsterdam. * The actaul wikitext is stored on Apache servers, again I think these are all in Florida. They are not split into clusters like the database servers AFAICT. * There are squids - caching servers - located in Florida, Korea and Amsterdam (these are not the same as the toolserver machines).
So in other words the whole wikitext for all wikis is not backed up anywhere except in database dumps. Scary.
the wub
On Mon, Dec 1, 2008 at 6:04 PM, the wub thewub.wiki@googlemail.com wrote:
So in other words the whole wikitext for all wikis is not backed up anywhere except in database dumps. Scary.
That's not true. There is a wikitext backup on the toolserver cluster. And as you state they are not divided into clusters they should contain the text of all wikis.
Bryan
wikimedia-l@lists.wikimedia.org