Xmldatadumps-l

xmldatadumps-l@lists.wikimedia.org

720 discussions

what's needed to fork
by Ariel T. Glenn 27 Sep '12

27 Sep '12

A discussion on the toolserver list brought up the question, once again, of what would be needed to fork the projects. Because some data is private we aren't going to be able to provide data for perfect copies, but the content can be preserved. The question is how close we can get. In particular I would like folks to think about how we can manage the user account issue. It would be very nice indeed if users could reclaim their accounts on a copy of the project, and yet we cannot provide any outside project a copy of the user table (which has email addresses and other useful bits in it). And many many users don't give an email address anyways. I'd like to hear proposals for how this could be handled. Wouldn't it be awesome if this could be done today, and Wikipedia editors could have editing privileges on copies of the project around the globe that provided different experimental features? Assuming of course that there were groups or organizations that wanted to run such copies of the site... Ariel

2 1

What is the best way to extract bzip2 and be able to parse the text
by Tubtim Eawchoowong 19 Sep '12

19 Sep '12

I am planning to decompress XML formatted bzip2 file by downloading the file using Java class URL. Then I plan to decompress it on the fly using Apache Ant and then parse to store in MySQL database. I am not sure if there are better ways to do that. Also is there a way to update my database without having to go through the whole process every time?

1 0

Re: [Xmldatadumps-l] [Wikitech-l] HTML wikipedia dumps: Could you please provide them, or make public the code for interpreting templates?
by Jeremy Baron 17 Sep '12

17 Sep '12

On Sun, Sep 9, 2012 at 6:34 PM, Roberto Flores <f.roberto.isc(a)gmail.com> wrote: > I have developed an offline Wikipedia, Wikibooks, Wiktionary, etc. app for > the iPhone, which does a somewhat decent job at interpreting the wiki > markup into HTML. > However, there are too many templates for me to program (not to mention, > it's a moving target). > Without converting these templates, many articles are simply unreadable and > useless. Templates are dumped just like all other pages are. Have you found them in the dumps? which dump are you looking at right now? > Could you please provide HTML dumps (I mean, with the templates > pre-processed into HTML, everything else the same as now) every 3 or 4 > months? 3 or 4 month frequency seems unlikely to be useful to many people. Otherwise no comment. > Or alternatively, could you make the template API available so I could > import it in my program? How would this template API function? What does import mean? -Jeremy

6 6

DEWIKI SQL dump, 3 queries
by Gregor Martynus 09 Sep '12

09 Sep '12

Hey there, can anybody with access execute these 3 SQL queries and provide me the results of it? --1 SELECT user_id, user_name, user_registration FROM user INNER JOIN logging ON log_user = user_id WHERE LEFT(user_registration, 4) = 2012 AND user_id NOT IN (SELECT ipb_user FROM ipblocks) AND log_type = 'newusers' AND log_action = 'create'; --2 SELECT page_id, page_title, page_namespace, page_is_redirect FROM page; --3 INSERT INTO u_hoo.dbq189 SELECT user_name FROM user INNER JOIN logging ON log_user = user_id WHERE LEFT(user_registration, 4) = 2012 AND user_id NOT IN (SELECT ipb_user FROM ipblocks) AND log_type = 'newusers' AND log_action = 'create'; SELECT rev_id, rev_page, rev_comment, rev_user_text, rev_user, rev_timestamp FROM revision INNER JOIN u_hoo.dbq189 ON rev_user_text = dbq189.user_name WHERE rev_deleted = 0 AND rev_user != 0; I need it for a study of a friend of mine, I really appreciate your help. I also made a ticket, but nobody reacted so far and it's a bit urgent: https://jira.toolserver.org/browse/DBQ-195 Thanks – Gregor

3 6

7zip on *-pages-articles
by Zak Wilcox 02 Sep '12

02 Sep '12

Hi dump producers, I know there's more to the choice of compression format than the size of the resulting dumps (e.g. time, memory, portability, existing code investment) and I read that you looked at LZMA and found it to be of insignificant benefit [1], but I noticed over at the Large Text Compression Benchmark site that they use 7-zip in PPMd mode and did some trial recompressions. The bzip dumps use 900k blocks and according to the bzip2.org implementation's manual it takes around 7600k while compressing and around 3700k while decompressing. Like LZMA, PPMd apparently uses the same amount of memory for decompression as it used during compression, so I recompressed the XML dump with various amounts of memory so you can make your own comparisons. Specifically, using 7zip 9.20 from Ubuntu Precise's p7zip-full, I ran: for MEM in 3700k 7600k 16m 512m; do bzcat enwiki-20120802-pages-articles.xml.bz2 \ | 7z -a -si -m0=PPMd:mem=$MEM \ enwiki-20120802-pages-articles.xml.$MEM.7z done bzcat enwiki-20120802-pages-articles.xml.bz2 \ | 7z -a -si enwiki-20120802-pages-articles.xml.LZMA.7z for the following resulting file sizes in bytes (% of .bz2 version): original bz2: 9143865996 $MEM=3700k : 8648303296 (94.6%) $MEM=7600k : 8043626528 (88.0%) $MEM=16m : 7910637814 (86.5%) (the default for both PPMd & LZMA) LZMA: 7705327210 (84.3%) $MEM=512m : 7076755355 (77.4%) I wasn't looking to compare running times and absolute values wouldn't compare to your servers but for what it's worth I noticed that LZMA took over twice as long as any PPMd run. I was expecting PPMd to beat LZMA, hence the several PPMd runs. There's probably some value in experimenting with PPMd's "model order" too, which I didn't try. Google "model order for PPMd" or see /usr/share/doc/p7zip-full/DOCS/MANUAL/switches/method.htm#PPMd (on Debian/Ubuntu). As the dump servers only have to do it once to save that bandwidth for every download from every mirror that month, perhaps it's worth giving 7zip more memory than bzip or even more than the default, although I appreciate that you drive some users out of the market if compression memory requirements equal decompression requirements and you start using a few gig to compress. Also while you can (with a little effort) seek around bz2s and extract individual blocks, PPMd's seekability isn't something I've explored. Just some thoughts. Zak [1]: "7-Zip's LZMA compression produces significantly smaller files for the full-history dumps, but doesn't do better than bzip2 for our other files." --- http://meta.wikimedia.org/wiki/Data_dumps

2 1

Dumb file of a certain flattened category
by tzali maimon 14 Aug '12

14 Aug '12

Hello, As I forwarded below, I am looking to produce a list of IDs of all articles in wikipedia under a certain category (including its sub-categories and sub-sub-categories etc.) For example: a list of IDs of all articles under Mathematics. I wodered weather or not you have a dump file that has that list per category. I know you have a dump file for ALL categories which weighs 8GB. I am looking for partial lists and wondered if you produce them. I know Semantic MediaWiki can faltten a CategoryTree but how can I get the info of the list in one textual file? Thanks for the answer and your help Tzali Maimon -- צאלי

1 0

Citizendium full XML dump: 168, 262 pages and 753, 651 revisions
by emijrp 10 Aug '12

10 Aug '12

Hi; I think this is the first time a full XML dump of Citizendium is publicly available[1] (CZ offers dumps but only the last revision for each article[2], and our previously efforts generated corrupted and incomplete dumps). It contains 168,262 pages and 753,651 revisions (9 GB, 99 MB in 7z). I think it may be useful for researchers, including quality analysis. It was generated using WikiTeam tools.[3] This is part of our task force to make backups of thousands of wikis around the Internet.[4] Regards, emijrp [1] http://archive.org/details/wiki-encitizendiumorg [2] http://en.citizendium.org/wiki/CZ:Downloads [3] http://code.google.com/p/wikiteam/ [4] http://code.google.com/p/wikiteam/wiki/AvailableBackups -- Emilio J. Rodríguez-Posada. E-mail: emijrp AT gmail DOT com Pre-doctoral student at the University of Cádiz (Spain) Projects: AVBOT <http://code.google.com/p/avbot/> | StatMediaWiki<http://statmediawiki.forja.rediris.es> | WikiEvidens <http://code.google.com/p/wikievidens/> | WikiPapers<http://wikipapers.referata.com> | WikiTeam <http://code.google.com/p/wikiteam/> Personal website: https://sites.google.com/site/emijrp/

2 2

Malware reported in mirror
by Kevin Day 12 Jul '12

12 Jul '12

We got an automated notice from one of the big search engines that both http://ftpmirror.your.org and http://dumps.wikimedia.your.org were hosting some unspecified malware. I've verified nothing on the mirror box itself is compromised from the best I can tell, which leaves them being unhappy with something that we're mirroring. I've started ClamAV scanning the whole public volume, but that's going to take quite a while (+20 million files, 80TB of data). The only thing it's complained about so far is: http://ftpmirror.your.org/pub/wikimedia/images/wiktionary/fj/c/c4/citibank-… which was making the scanner crash. I don't see anything wrong with the file itself though. Is it possible someone could have uploaded something at one point that was malicious and it's still floating around in the archives that got pushed to us? -- Kevin

4 8

Re: [Xmldatadumps-l] anonymous user account logs (account created / account blocked)
by Platonides 03 Jul '12

03 Jul '12

On 02/07/12 18:45, Gregor Martynus wrote: > Hey there, > > I really appreciate the help of you and the others in the mailing list, > thanks to your help I was able to extract the data my friend needs for > her wikipedia study. > > While it worked as a proof of concept, that the data she needs is > available, it took to much time to process all needed data and she needs > it another 3 to 4 times. > > I contact you because you offered me to send an SQL dump back then, does > that mean you've access to the Wikipedia database? > > I'm not an SQL expert, but I think based on the MediaWiki database > schema, I could build the SQL queries for the 3 things she needs > > 1. user_accounts(name, created_at, blocked_at) created manually in 2012 > 2. pages (namespace_id, title, is_redirect) > 3. revisions (page_id, user_id, created_at) created by users from 1. > bytes_diff would be nice, if available > > The goal is to find users that manually created an account, have not > been blocked and that made their first contribution in June 2012. > > Any chance you could run the queries and send me the results as sql > dumps? Or if not, do you know somebody I could ask? > > Thanks again for your help! > > -- > Gregor Martynus Given that your requisites here are more specific than those of the bug, I have made the following query for you, taking advantage that the recentchanges talbe keeps the data from last month: select rc_timestamp, rc_user_text, user_registration, rc_namespace, rc_title, rc_old_len, rc_new_len, rc_new, rc_minor, rc_type, page_is_redirect from recentchanges JOIN user on (user_id=rc_user) LEFT JOIN page ON (page_id=rc_cur_id) where rc_timestamp > '201206000000' and user_registration LIKE '2012%' order by rc_timestamp asc Some notes: - This will contain from June 2 to July 2. - No bytes_diff, but you have old and new byte len :) - It may have some non-edit log entries. File lives at http://toolserver.org/~platonides/sandbox/martynus.txt I also made a list of the accounts created in 2012 which were made manually: SELECT user_id, user_name, log_timestamp AS signed_up_at FROM user LEFT JOIN logging ON user_id = log_user WHERE log_type = 'newusers' AND log_action = 'create' AND log_timestamp > '20120000000000 File lives at http://toolserver.org/~platonides/sandbox/martynus2.txt And of block actions done to users registered in 2012: SELECT log_action, log_timestamp, log_title, user_registration FROM logging JOIN user on (log_title=user_name) WHERE log_type = 'block'AND log_timestamp > '20120000000000' and user_registration > '20120000000000' File lives at http://toolserver.org/~platonides/sandbox/martynus3.txt I guess that will work for your needs. Regards PS: Rows per file: 73426 martynus.txt 54429 martynus2.txt 2893 martynus3.txt

2 1

en wiki dumps halted while...
by Ariel T. Glenn 03 Jul '12

03 Jul '12

db lag on one of the en wiki slaves is investigated; don't know how long it will take to find the cause or fix it (not days, but surely hours). Ariel

1 1

← Newer
1
...
51
52
53
54
55
56
57
...
72
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Xmldatadumps-l