[Foundation-l] EN Wikipedia Editing Statistics

List overview All Threads
Download

newer

older

Re: [Foundation-l] foundation-l...

[Foundation-l] EN Wikipedia...

Erik Zachte

30 Nov 2008 30 Nov '08

5:20 p.m.

English -> English dump

...

Because myself and others have been frustrated by the lack of good stats on the number of active editors on the English Wikipedia, I have compiled some stats on the editing frequency on enwiki:

No worries: in only 176 days from now the English dump will be ready and I can run wikistats scripts on it. It just started 52 days ago, so let us be patient for a while ;) http://www.infodisiac.com/cgi-bin/WikimediaDownload.pl Nice stats though! Erik Zachte

Show replies by date

Robert Rohde

30 Nov 30 Nov

9:11 p.m.

On Sun, Nov 30, 2008 at 8:20 AM, Erik Zachte <erikzachte(a)infodisiac.com> wrote:

...

English -> English dump

Because myself and others have been frustrated by the lack of good stats on the number of active editors on the English Wikipedia, I have compiled some stats on the editing frequency on enwiki:

No worries: in only 176 days from now the English dump will be ready and I can run wikistats scripts on it. It just started 52 days ago, so let us be patient for a while ;)

Is there any reason at all to believe that it is more likely to finish this time than all the previous attempts during the last two years? I have virtually zero faith in a script that takes 230 days and where any error wipes out all progress. -Robert Rohde

Michael Peel

9:35 p.m.

On 30 Nov 2008, at 20:11, Robert Rohde wrote:

...

On Sun, Nov 30, 2008 at 8:20 AM, Erik Zachte <erikzachte(a)infodisiac.com> wrote:

English -> English dump

Because myself and others have been frustrated by the lack of good stats on the number of active editors on the English Wikipedia, I have compiled some stats on the editing frequency on enwiki:

No worries: in only 176 days from now the English dump will be ready and I can run wikistats scripts on it. It just started 52 days ago, so let us be patient for a while ;)

Hold on...what? There is no recent dump of the English Wikipedia, and there hasn't been for the last 2 years? Please tell me I'm misunderstanding things here. Mike

Chad

9:51 p.m.

On Sun, Nov 30, 2008 at 3:35 PM, Michael Peel <email(a)mikepeel.net> wrote:

...

On 30 Nov 2008, at 20:11, Robert Rohde wrote:

On Sun, Nov 30, 2008 at 8:20 AM, Erik Zachte <erikzachte(a)infodisiac.com> wrote:

English -> English dump

Because myself and others have been frustrated by the lack of good stats on the number of active editors on the English Wikipedia, I have compiled some stats on the editing frequency on enwiki:

No worries: in only 176 days from now the English dump will be ready and I can run wikistats scripts on it. It just started 52 days ago, so let us be patient for a while ;)

Hold on...what? There is no recent dump of the English Wikipedia, and there hasn't been for the last 2 years? Please tell me I'm misunderstanding things here. Mike _______________________________________________ foundation-l mailing list foundation-l(a)lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

(cc'd to wikitech-l) I saw this the other day as well and found it odd. While enwiki dumps do take the longest, this does seem like an _incredibly_ long time for "All pages with complete page edit history (.bz2)" to finish (May 2009). -Chad

Thomas Dalton

9:58 p.m.

...

I saw this the other day as well and found it odd. While enwiki dumps do take the longest, this does seem like an _incredibly_ long time for "All pages with complete page edit history (.bz2)" to finish (May 2009).

Do you know how many pages enwiki has and how much edit history they each have? It's a lot! I think the dumps work by starting with the last successful dump and just adding in anything that's changed, but because there haven't been any successful dumps of the whole of enwiki in a long time, it basically has to start from scratch, which is going to take a long time (and means it probably won't succeed - ie. we have a catch-22). It seems to me that (if my understanding of the problem is correct), the answer is to devote a more powerful computer to the dump for just this one so that we can get things moving again - I'm sure if we asked around someone could lend us a really powerful computer for a few weeks to do the dump on.

Robert Rohde

10:20 p.m.

On Sun, Nov 30, 2008 at 12:58 PM, Thomas Dalton <thomas.dalton(a)gmail.com> wrote:

...

No, dumps are total, not incremental. It is really more than throwing a big computer at it. The dumping process ought to be redesigned to be more fault tolerant and faster. It is ridiculous to have a process that is expected to take months and yet have no method of saving one's progress as it goes and restarting it in case of trouble. -Robert Rohde

Thomas Dalton

11:39 p.m.

...

No, dumps are total, not incremental.

The dumps are, yes, but what about the creation of the dumps? My understanding was that they are created by taking an old dump and adding to it.

Robert Rohde

1 Dec 1 Dec

12:02 a.m.

On Sun, Nov 30, 2008 at 2:39 PM, Thomas Dalton <thomas.dalton(a)gmail.com> wrote:

...

No, dumps are total, not incremental.

The dumps are, yes, but what about the creation of the dumps? My understanding was that they are created by taking an old dump and adding to it.

No. Each dump process starts fresh from the database, they don't add to old dumps. (Or at least they certainly didn't the last time I looked at the dumping scripts, and I have no reason to suspect that has changed.) -Robert

geni

30 Nov 30 Nov

9:56 p.m.

2008/11/30 Michael Peel <email(a)mikepeel.net>et>:

...

Hold on...what? There is no recent dump of the English Wikipedia, and there hasn't been for the last 2 years? Please tell me I'm misunderstanding things here. Mike

Not to years but yes the English Wikipedia dumps very rarely work. De.wikipedia is starting to suffer the same issues and image database dumps don't happen. -- geni

Robert Rohde

10:23 p.m.

On Sun, Nov 30, 2008 at 12:56 PM, geni <geniice(a)gmail.com> wrote:

...

2008/11/30 Michael Peel <email(a)mikepeel.net>et>:

Hold on...what? There is no recent dump of the English Wikipedia, and there hasn't been for the last 2 years? Please tell me I'm misunderstanding things here. Mike

Not to years but yes the English Wikipedia dumps very rarely work. De.wikipedia is starting to suffer the same issues and image database dumps don't happen.

No, I think it really has been two years (September 2006 to be precise). I'm pretty sure there have been no complete dumps this calendar year. I believe there was one (or two?) full dump process that claimed to run to completion in 2007 but it was later found to have been truncated (i.e. it didn't really dump all of enwiki, only a portion of it). If I am wrong and there really is a more recent complete history dump of enwiki floating around somewhere, then I'd love to hear about it, but I don't believe that is the case. -Robert Rohde

Thomas Dalton

11:38 p.m.

...

Not to years but yes the English Wikipedia dumps very rarely work. De.wikipedia is starting to suffer the same issues and image database dumps don't happen.

Neil Harris

1 Dec 1 Dec

3:16 a.m.

Thomas Dalton wrote:

...

Not to years but yes the English Wikipedia dumps very rarely work. De.wikipedia is starting to suffer the same issues and image database dumps don't happen.

That sounds about right to me. I think the confusion may come from there being lots of different dumps - the smaller dumps of enwiki do succeed (occasionally, at least!), it's the full dump of every revision of every page that fails routinely. _______________________________________________ foundation-l mailing list foundation-l(a)lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

Is the data replicated anywhere outside the Tampa data centre (such as in Amsterdam or Seoul)? If not, just one fire, flood or hurricane could destroy the entire en: Wikipedia. -- Neil

geni

3:19 a.m.

2008/12/1 Neil Harris <usenet(a)tonal.clara.co.uk>uk>:

...

Is the data replicated anywhere outside the Tampa data centre (such as in Amsterdam or Seoul)? If not, just one fire, flood or hurricane could destroy the entire en: Wikipedia.

En probably isn't that bad. We have lost commons images already. -- geni

Robert Rohde

4:09 a.m.

On Sun, Nov 30, 2008 at 6:16 PM, Neil Harris <usenet(a)tonal.clara.co.uk> wrote:

...

Is the data replicated anywhere outside the Tampa data centre (such as in Amsterdam or Seoul)? If not, just one fire, flood or hurricane could destroy the entire en: Wikipedia.

There are database mirrors of every wiki, including en, as part of the toolserver cluster in Amsterdam. -Robert Rohde

teun spaans

5:04 a.m.

It is good to hear that en: is replicated. I assume this also applies to commons? Still, it might be a good idea to think about a redesign of the dump process. On Mon, Dec 1, 2008 at 4:09 AM, Robert Rohde <rarohde(a)gmail.com> wrote:

...

On Sun, Nov 30, 2008 at 6:16 PM, Neil Harris <usenet(a)tonal.clara.co.uk> wrote:

Is the data replicated anywhere outside the Tampa data centre (such as in Amsterdam or Seoul)? If not, just one fire, flood or hurricane could destroy the entire en: Wikipedia.

There are database mirrors of every wiki, including en, as part of the toolserver cluster in Amsterdam. -Robert Rohde _______________________________________________ foundation-l mailing list foundation-l(a)lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

Robert Rohde

6:21 a.m.

On Sun, Nov 30, 2008 at 8:04 PM, teun spaans <teun.spaans(a)gmail.com> wrote:

...

It is good to hear that en: is replicated. I assume this also applies to commons?

The database yes. For images there is as-needed caching in Amsterdam and Korea, but my understanding is that the only complete copies of the file store all live in Florida. On a related point, replicated mirrors deal well enough with acts of God (e.g. if a meteor hits the data center in Florida), but because everything is live mirrored it doesn't deal well if the server is accidentally instructed to delete something. That erroneous instruction would be carried to all the mirrors as well, and you might really lose something before the links could be severed. We really need to have periodic stored backups in addition to the replicated mirrors. -Robert Rohde

Nikola Smolenski

7:29 a.m.

On Monday 01 December 2008 04:09:11 Robert Rohde wrote:

...

On Sun, Nov 30, 2008 at 6:16 PM, Neil Harris <usenet(a)tonal.clara.co.uk>

wrote:

...

Is the data replicated anywhere outside the Tampa data centre (such as in Amsterdam or Seoul)? If not, just one fire, flood or hurricane could destroy the entire en: Wikipedia.

There are database mirrors of every wiki, including en, as part of the toolserver cluster in Amsterdam.

Unfortunately, enwiki mirror doesn't include article text :(

Robert Rohde

7:50 a.m.

On Sun, Nov 30, 2008 at 10:29 PM, Nikola Smolenski <smolensk(a)eunet.yu> wrote:

...

On Monday 01 December 2008 04:09:11 Robert Rohde wrote:

On Sun, Nov 30, 2008 at 6:16 PM, Neil Harris <usenet(a)tonal.clara.co.uk>

wrote:

Is the data replicated anywhere outside the Tampa data centre (such as in Amsterdam or Seoul)? If not, just one fire, flood or hurricane could destroy the entire en: Wikipedia.

There are database mirrors of every wiki, including en, as part of the toolserver cluster in Amsterdam.

Unfortunately, enwiki mirror doesn't include article text :(

Ouch, I hadn't realized they gave up on text replication. Apparently quite a while ago too. (That will teach me for never bothering to learn to use the toolserver.) So I guess we are back to the meteor impact destroys Wikipedia scenario. -Robert Rohde

Neil Harris

12:18 p.m.

Robert Rohde wrote:

...

On Sun, Nov 30, 2008 at 10:29 PM, Nikola Smolenski <smolensk(a)eunet.yu> wrote:

On Monday 01 December 2008 04:09:11 Robert Rohde wrote:

On Sun, Nov 30, 2008 at 6:16 PM, Neil Harris <usenet(a)tonal.clara.co.uk>

wrote:

Is the data replicated anywhere outside the Tampa data centre (such as in Amsterdam or Seoul)? If not, just one fire, flood or hurricane could destroy the entire en: Wikipedia.

There are database mirrors of every wiki, including en, as part of the toolserver cluster in Amsterdam.

Unfortunately, enwiki mirror doesn't include article text :(

That's scary: are there any off-site backups of the full database, including the article text, made anywhere, other than the constantly failing dumps? Given that Wikipedia is the Wikimedia Foundation's principal asset, I would hope that fixing this single point of failure would be a priority for the Foundation. -- Neil

Bryan Tong Minh

1:08 p.m.

On Mon, Dec 1, 2008 at 7:29 AM, Nikola Smolenski <smolensk(a)eunet.yu> wrote:

...

On Monday 01 December 2008 04:09:11 Robert Rohde wrote:

On Sun, Nov 30, 2008 at 6:16 PM, Neil Harris <usenet(a)tonal.clara.co.uk>

wrote:

Is the data replicated anywhere outside the Tampa data centre (such as in Amsterdam or Seoul)? If not, just one fire, flood or hurricane could destroy the entire en: Wikipedia.

There are database mirrors of every wiki, including en, as part of the toolserver cluster in Amsterdam.

Unfortunately, enwiki mirror doesn't include article text :(

Are you sure about that? Last time I checked the text databases were shared between all wikimedia project and thus replicated all at once or not at all. Bryan

the wub

6:04 p.m.

2008/12/1 Bryan Tong Minh <bryan.tongminh(a)gmail.com>om>:

...

On Mon, Dec 1, 2008 at 7:29 AM, Nikola Smolenski <smolensk(a)eunet.yu> wrote:

On Monday 01 December 2008 04:09:11 Robert Rohde wrote:

On Sun, Nov 30, 2008 at 6:16 PM, Neil Harris <usenet(a)tonal.clara.co.uk>

wrote:

Is the data replicated anywhere outside the Tampa data centre (such as in Amsterdam or Seoul)? If not, just one fire, flood or hurricane could destroy the entire en: Wikipedia.

There are database mirrors of every wiki, including en, as part of the toolserver cluster in Amsterdam.

Unfortunately, enwiki mirror doesn't include article text :(

Are you sure about that? Last time I checked the text databases were shared between all wikimedia project and thus replicated all at once or not at all. Bryan

http://upload.wikimedia.org/wikipedia/commons/f/ff/Wikimedia-servers-2008-1… is a good picture of it. If I'm interpreting it correctly: * There are the "database servers" running MySQL, which store metadata. These are split into clusters (en on 1; de + commons + possibly others on 2; others on 3). I believe these are all located in Florida. * These databases (i.e. only metadata) are replicated for the Toolserver machines located in Amsterdam. * The actaul wikitext is stored on Apache servers, again I think these are all in Florida. They are not split into clusters like the database servers AFAICT. * There are squids - caching servers - located in Florida, Korea and Amsterdam (these are not the same as the toolserver machines). So in other words the whole wikitext for all wikis is not backed up anywhere except in database dumps. Scary. the wub

Bryan Tong Minh

2 Dec 2 Dec

10:11 a.m.

On Mon, Dec 1, 2008 at 6:04 PM, the wub <thewub.wiki(a)googlemail.com> wrote:

...

So in other words the whole wikitext for all wikis is not backed up anywhere except in database dumps. Scary.

That's not true. There is a wikitext backup on the toolserver cluster. And as you state they are not divided into clusters they should contain the text of all wikis. Bryan

5624

days inactive

5626

days old

wikimedia-l@lists.wikimedia.org

Manage subscription

21 comments

11 participants

tags (0)

participants (11)

Bryan Tong Minh
Chad
Erik Zachte
geni
Michael Peel
Neil Harris
Nikola Smolenski
Robert Rohde
teun spaans
the wub
Thomas Dalton