Dumps - Toolserver-l - lists.wikimedia.org

List overview All Threads
Download

Dumps

OpenStreetMap on the toolserver...

An issue

John

5 May 2012 5 May '12

7:24 p.m.

I was taking a look at our dumps in user-store and none of them are compressed, and I was socked about that. I know a lot of people use pywikipedia to parse the dumps, and I know it can handle the bz2 files. any reason we dont just make them all bz2? John

Attachments:

attachment.htm (text/html — 268 bytes)

Reply

Show replies by date

Hydriz Wikipedia

5 May 5 May

7:27 p.m.

Yep, if we can use compressed dumps, we can use much lesser resources that what it is using now (currently >800GB). On Sun, May 6, 2012 at 12:24 AM, John <phoenixoverride(a)gmail.com> wrote:

I was taking a look at our dumps in user-store and none of them are compressed, and I was socked about that. I know a lot of people use pywikipedia to parse the dumps, and I know it can handle the bz2 files. any reason we dont just make them all bz2? John _______________________________________________ Toolserver-l mailing list (Toolserver-l(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette

-- Regards, Hydriz We've created the greatest collection of shared knowledge in history. Help protect Wikipedia. Donate now: http://donate.wikimedia.org

Reply

Federico Leva (Nemo)

19 Jul 19 Jul

10:33 p.m.

We don't use only PWB. Danny_B made some work recently so that all dumps are in the same place etc., what consumed the most space were duplicates. This was supposed to be documented on https://wiki.toolserver.org/view/User-store but it's never been. Most space is taken by view stats whose compression is discussed here: https://wiki.toolserver.org/view/Talk:User-store Nemo

Reply

Danny B.

21 Jul 21 Jul

7:53 p.m.

We don't use only PWB. Danny_B made some work recently so that all dumps are in the same place etc., what consumed the most space were duplicates. This was supposed to be documented on https://wiki.toolserver.org/view/User-store but it's never been. Most space is taken by view stats whose compression is discussed here: https://wiki.toolserver.org/view/Talk:User-store

There is some basic doc in "readme" files in /mnt/usr-store[/dumps[/...]] though. However, thanks for reminding me the wiki doc, I'll try to put it on ASAP. Danny B.

Reply

4290

days inactive

4367

days old

toolserver-l@lists.wikimedia.org

Manage subscription

3 comments

4 participants

tags (0)

participants (4)

Danny B.
Federico Leva (Nemo)
Hydriz Wikipedia
John