Wikipedia dumps downloader

List overview All Threads
Download

newer

older

Re: [Xmldatadumps-l]...

Re: [Xmldatadumps-l] Wikipedia...

emijrp

26 Jun 2011 26 Jun '11

12:53 p.m.

Hi all; Can you imagine a day when Wikipedia is added to this list?[1] WikiTeam have developed a script[2] to download all the Wikipedia dumps (and her sister projects) from dumps.wikimedia.org. It sorts in folders and checks md5sum. It only works on Linux (it uses wget). You will need about 100GB to download all the 7z files. Save our memory. Regards, emijrp [1] http://en.wikipedia.org/wiki/Destruction_of_libraries [2] http://code.google.com/p/wikiteam/source/browse/trunk/wikipediadownloader.py

Attachments:

attachment.htm (text/html — 757 bytes)

Show replies by thread

Samuel Klein

27 Jun 27 Jun

10:18 a.m.

New subject: [Wiki-research-l] Wikipedia dumps downloader

Thank you, Emijrp! What about the dump of Commons images? [for those with 10TB to spare] SJ On Sun, Jun 26, 2011 at 8:53 AM, emijrp <emijrp(a)gmail.com> wrote:

...

-- Samuel Klein identi.ca:sj w:user:sj +1 617 529 4266

emijrp

11:10 a.m.

New subject: [Wiki-research-l] Wikipedia dumps downloader

Hi SJ; You know that that is an old item in our TODO list ; ) I heard that Platonides developed a script for that task long time ago. Platonides, are you there? Regards, emijrp 2011/6/27 Samuel Klein <sjklein(a)hcs.harvard.edu>

...

Thank you, Emijrp! What about the dump of Commons images? [for those with 10TB to spare] SJ On Sun, Jun 26, 2011 at 8:53 AM, emijrp <emijrp(a)gmail.com> wrote:

Hi all; Can you imagine a day when Wikipedia is added to this list?[1] WikiTeam have developed a script[2] to download all the Wikipedia dumps

(and

her sister projects) from dumps.wikimedia.org. It sorts in folders and checks md5sum. It only works on Linux (it uses wget). You will need about 100GB to download all the 7z files. Save our memory. Regards, emijrp [1] http://en.wikipedia.org/wiki/Destruction_of_libraries [2]

http://code.google.com/p/wikiteam/source/browse/trunk/wikipediadownloader.py

_______________________________________________ Wiki-research-l mailing list Wiki-research-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

-- Samuel Klein identi.ca:sj w:user:sj +1 617 529 4266 _______________________________________________ Wiki-research-l mailing list Wiki-research-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

lior gimel

1:48 p.m.

New subject: [Wiki-research-l] Wikipedia dumps downloader

Hi, Does anyone know if old dumps, from around 2004-2005, are still available somewhere? Lior On Mon, Jun 27, 2011 at 2:10 PM, emijrp <emijrp(a)gmail.com> wrote:

...

Thank you, Emijrp! What about the dump of Commons images? [for those with 10TB to spare] SJ On Sun, Jun 26, 2011 at 8:53 AM, emijrp <emijrp(a)gmail.com> wrote:

Hi all; Can you imagine a day when Wikipedia is added to this list?[1] WikiTeam have developed a script[2] to download all the Wikipedia dumps

(and

http://code.google.com/p/wikiteam/source/browse/trunk/wikipediadownloader.py

_______________________________________________ Wiki-research-l mailing list Wiki-research-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

Samuel Klein

2:16 p.m.

New subject: [Foundation-l] [Wiki-research-l] Wikipedia dumps downloader

On Mon, Jun 27, 2011 at 7:10 AM, emijrp <emijrp(a)gmail.com> wrote:

...

Hi SJ; You know that that is an old item in our TODO list ; )

I know, I know... I don't mean to Domas by asking after the Commons dump every time dumps of any kind comes up. It's just so inspiring to imagine that consolidated visual and auditory beauty being mirrored all around the world, it is difficult to resist. SJ.

...

I heard that Platonides developed a script for that task long time ago. Platonides, are you there? Regards, emijrp 2011/6/27 Samuel Klein <sjklein(a)hcs.harvard.edu>

Thank you, Emijrp! What about the dump of Commons images? [for those with 10TB to spare] SJ On Sun, Jun 26, 2011 at 8:53 AM, emijrp <emijrp(a)gmail.com> wrote:

Hi all; Can you imagine a day when Wikipedia is added to this list?[1] WikiTeam have developed a script[2] to download all the Wikipedia dumps

(and

http://code.google.com/p/wikiteam/source/browse/trunk/wikipediadownloader.py

_______________________________________________ Wiki-research-l mailing list Wiki-research-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

_______________________________________________ foundation-l mailing list foundation-l(a)lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

-- Samuel Klein identi.ca:sj w:user:sj +1 617 529 4266

Platonides

9:32 p.m.

New subject: [Wiki-research-l] Wikipedia dumps downloader

emijrp wrote:

...

Hi SJ; You know that that is an old item in our TODO list ; ) I heard that Platonides developed a script for that task long time ago. Platonides, are you there? Regards, emijrp

Yes, I am. :)

emijrp

28 Jun 28 Jun

7:50 a.m.

New subject: [Wiki-research-l] Wikipedia dumps downloader

Can you share your script with us? 2011/6/27 Platonides <platonides(a)gmail.com>

...

emijrp wrote:

Hi SJ; You know that that is an old item in our TODO list ; ) I heard that Platonides developed a script for that task long time ago. Platonides, are you there? Regards, emijrp

Yes, I am. :)

4710

days inactive

4712

days old

xmldatadumps-l@lists.wikimedia.org

Manage subscription

6 comments

5 participants

tags (0)

participants (5)

emijrp
lior gimel
Platonides
Samuel Klein
Samuel Klein