Xmldatadumps-l

xmldatadumps-l@lists.wikimedia.org

720 discussions

Re: [Xmldatadumps-l] Template expansion inconsistency
by Federico Leva (Nemo) 23 Feb '14

23 Feb '14

wp mirror, 23/02/2014 15:26: > c) Third best, would be to patch `mwxml2sql'. This I also favor, but > would like some guidance from its author, Ariel Glenn, before I start > hacking. This seems the most likely. Probably, mwxml2sql has to be fixed so that it does whatever importDump.php/Special:Import do. Only if they both have the same problem with full page names in <title>, then the export should be changed. This is my guess only; at any rate do file a bug if there is a difference in behaviour. Nemo

1 0

Fwd: Template expansion inconsistency
by wp mirror 23 Feb '14

23 Feb '14

Dear Nemo, Thanks for enlightening me regarding <title>. I did not know that it was intended to be a compound of namespace word and `page_title' field. Still, I have some thoughts on this matter. 1) importDump.php As of WP-MIRROR 0.6, `importDump.php' is not longer used. The disadvantage of `importDump.php' is that it is slow. Importation of `enwiki' takes about two months, which is greater than the interval between XML dumps. The advantage of `importDump.php' is that it handles any idiosyncrasy (such as compound <title> entries) in the XML dumps. 2) mwxml2sql As of WP-MIRROR 0.6, `mwxml2sql' is used to convert the XML dump into a set of SQL dumps (for the `page', `revision', `text' tables) which can then be directly loaded into the underlying database tables. The advantage of `mwxml2sql' is that it is very fast. And, when used in conjunction with MySQL 5.5 fast index creation, one can load `enwiki' using 80% less time. The disadvantage is that it faithfully copies the <title> field into the SQL statement for INSERTing the `page_title' field. We now know that this results in pages from the Template and other namespaces being not found by MediaWiki, which then renders them as red-links. 3) First Normal Form One issue in the back of my mind concerns the recent changes in the XML schema. As of `export-0.6.xsd.gz' we note that ``Version 0.6 adds a separate namespace tag''. To my mind, the presence of the <ns> field should obviate the need to include a namespace word (e.g. `Category:', `Template:', etc.) within the <title> field. The principle is known as first normal form (1NF) which basically means that the contents of a field should be atomic rather than compound. 4) Solution Granted that the objective is to faithfully mirror the WMF database tables; the issue before us is this: Where along the tool chain should the patch be made. a) My instinct is to correct the issue upstream (the XML dump generation phase). The WMF `page_namespace' field should be copied to the <ns> field. The WMF `page_title' field should be copied to the <title> field. Adhere to principles of database normalization. b) Second best, would be to patch WP-MIRROR 0.7 to normalize the XML dump prior to feeding it into `mwxml2sql'. This I have done. c) Third best, would be to patch `mwxml2sql'. This I also favor, but would like some guidance from its author, Ariel Glenn, before I start hacking. d) A last resort would be to write an SQL query to clean up compound `page_title' entries in the mirror's database. But I really would rather not load unnormalized data in the first place. Sincerely Yours, Kent On 2/22/14, Federico Leva (Nemo) <nemowiki(a)gmail.com> wrote: > wp mirror, 22/02/2014 23:40: >> Still, it would be nice if the dump files could be fixed. > > Fixed? <title> is the full page name as it's supposed to be. Either > you're doing something wrong with the import, or the import > script/special page has a bug (not uncommon, but needs a bug report with > steps to reproduce). I see nothing to blame on the export side. > > Nemo > > _______________________________________________ > Xmldatadumps-l mailing list > Xmldatadumps-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l >

1 0

Incremental dump generator does not work
by Ivan 23 Feb '14

23 Feb '14

Hi, incremental dumps generator stopped working two days ago. Folders: http://dumps.wikimedia.org/other/incr/wikidatawiki/20140221/ http://dumps.wikimedia.org/other/incr/wikidatawiki/20140222/ are empty. Could somebody help with it? Sincerely, Ivan A. Krestinin

1 0

Template expansion inconsistency
by wp mirror 22 Feb '14

22 Feb '14

Dear Sir or Madam, I am not sure to which person or list I should address this question to. 0) Objective I am in the process of building DEB packages for: WP-MIRROR 0.7, the latest development version of MediaWiki 1.23, and a set of MediaWiki extensions. The objective is to this: That a page rendered by a mirror should look the same a that page rendered by the WMF site. 1) Problem In the process of testing mirrors, I noticed that many templates were not expanding, and instead being rendered as red-links. 2) Example To illustrate, consider the Ndash template, which appears on many pages such as <http://simple.wikipedia.org/wiki/August>. It appears in the underlying database: mysql> select page_id,page_title,rev_len,old_text from simplewiki.page,simplewiki.revision,simplewiki.text where page_id=rev_page and rev_text_id=old_id and page_title like 'Template:Ndash' limit 10\G *************************** 1. row *************************** page_id: 132985 page_title: Template:Ndash rev_len: 65 old_text: –<noinclude> [[Category:Formatting templates]] </noinclude> 1 row in set (0.25 sec) 3) Special:ExpandTemplates To test the above example ``Template:Ndash'', I use Special:ExpandTemplates. 3.1) Input text Today is the {{CURRENTDAY}} day.</br> This server is {{SERVER}}, script path {{SCRIPTPATH}}, current MW version {{CURRENTVERSION}}.</br> This site is {{SITENAME}}. Full page name is {{FULLPAGENAME}}.</br> <table> <tr><th>Template</th><th>Expanded</th><th>page_id</th><th>rev_len</th></tr> <tr><td>Ndash</td><td>{{Ndash}}</td><td>{{PAGEID: Ndash}}</td><td>{{PAGESIZE: Ndash}}</td></tr> <tr><td>Template:Ndash</td><td>{{Template:Ndash}}</td> <td>{{PAGEID: Template:Ndash}}</td><td>{{PAGESIZE: Template:Ndash}}</td></tr> <tr><td>Template:Template:Ndash</td><td>{{Template:Template:Ndash}}</td> <td>{{PAGEID: Template:Template:Ndash}}</td><td>{{PAGESIZE: Template:Template:Ndash}}</td></tr> </table> 3.2) <http://simple.wikipedia.site/wiki/Special:ExpandTemplates> Preview Here is the result from the WMF site: Today is the 21 day. This server is //simple.wikipedia.org, script path /w, current MW version 1.23wmf14 (f8b9201). This site is Wikipedia. Full page name is My template. Template Expanded page_id rev_len Ndash – 0 0 Template:Ndash – 132985 65 Template:Template:Ndash Template:Template:Ndash 0 0 Both {{Ndash}} and {{Template:Ndash}} expand as expected. 3.3) <http://simple.wikipedia.site/wiki/Special:ExpandTemplates> Preview Here is the result from the mirrored site: Today is the 21 day. This server is http://simple.wikipedia.site, script path /w, current MW version 1.23alpha. This site is simplewiki. Full page name is My template. Template Expanded page_id rev_len Ndash Template:Ndash 0 0 Template:Ndash Template:Ndash 0 0 Template:Template:Ndash – 132985 65 Only {{Template:Template:Ndash}} expands! 4) Question Why do I need to prepend an extra ``Template:'' to make the templates work for the mirror? Better yet: Could someone tell me where in the MediaWiki core I can find the code that takes the template (e.g. {{Ndash}} or {{Template:Ndash}}) and converts it into an SQL query that SELECTs the template expansion from the underlying database? Sincerely Yours, Kent

3 4

Updating mwxml2sql-0.0.2 for mediawiki 1.23 LTS
by wp mirror 05 Feb '14

05 Feb '14

Dear Ariel, I have been reading your code for `mwxml2sql-0.0.2' with a view towards updating it for mediawiki-1.23 LTS. 0) Support status Currently, the version info for `mwxml2sql' states the following: (shell)$ mwxml2sql --version mwxml2sql 0.0.2 Supported input schema versions: 0.4 through 0.8. Supported output MediaWiki versions: 1.5 through 1.21. 1) Current input schema version Currently, your XML dump files have the following header: (shell)$ head -n 1 zuwiki-20140121-pages-articles.xml <mediawiki xmlns="http://www.mediawiki.org/xml/export-0.8/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.mediawiki.org/xml/export-0.8/ http://www.mediawiki.org/xml/export-0.8.xsd" version="0.8" xml:lang="zu"> >From this I gather that XML schema is still 0.8, and that `mwxml2sql' needs no update on that head. 2) Current output MediaWiki version I reviewed the database schema for the `page', `revision', and `text' tables: <https://www.mediawiki.org/wiki/Manual:Page_table>, <https://www.mediawiki.org/wiki/Manual:Revision_table>, and <https://www.mediawiki.org/wiki/Manual:Text_table> It appears that the most recent changes to the schema for these three tables occurred for mediawiki versions 1.21, 1.21, and 1.19, respectively. >From this I gather that the database schema used for mediawiki 1.23 LTS is the same as that used for mediawiki 1.21; and therefore `mwxml2sql' needs no update on that head. 3) Recommended updates >From a review of your code, I concluded that two minor changes would be useful. 3.1) mwxml2sql.c The following three lines: (shell)$ grep 21 mwxml2sql.c fprintf(stderr,"Supported output MediaWiki versions: 1.5 through 1.21.\n\n"); /* we know MW 1.5 through MW 1.21 even though there is no MW 1.21 yet */ if (mwv->major != 1 || mwv->minor < 5 || mwv->minor > 21) { should read fprintf(stderr,"Supported output MediaWiki versions: 1.5 through 1.23.\n\n"); /* we know MW 1.5 through MW 1.23 */ if (mwv->major != 1 || mwv->minor < 5 || mwv->minor > 23) { 3.2) mwxmlelts.c The following line: (shell)$ grep 21 mwxmlelts.c <generator>MediaWiki 1.21wmf6</generator> should read <generator>MediaWiki 1.23wmf10</generator> 4) Request Please let me know if you agree with the above assessment. If you do, I would be happy to submit the changes to <https://gerrit.wikimedia.org/> for review. Sincerely Yours, Kent

1 0

abstract dumps broken, dumps paused til fixed
by Ariel T. Glenn 03 Feb '14

03 Feb '14

A recent update to the mediawiki multiversion scripts broke the abstract dumps; a bug report and a fix have been submitted so I expect this to get taken care of by Monday at the latest and hopefully over the weekend. In the meantime no new jobs for small wikis will be produced; I'll start those up again once the fix is in, as well as rerunning the abstract dumps where it failed. Currently running jobs will run to completion. Ariel

1 1

Re: [Xmldatadumps-l] (no subject)
by Randall Farmer 21 Jan '14

21 Jan '14

> For external uses like XML dumps integrating the compression > strategy into LZMA would however be very attractive. This would also > benefit other users of LZMA compression like HBase. For dumps or other uses, 7za -mx=3 / xz -3 is your best bet. That has a 4 MB buffer, compression ratios within 15-25% of current 7zip (or histzip), and goes at 30MB/s on my box, which is still 8x faster than the status quo (going by a 1GB benchmark). Trying to get quick-and-dirty long-range matching into LZMA isn't feasible for me personally and there may be inherent technical difficulties. Still, I left a note on the 7-Zip boards as folks suggested; feel free to add anything there: https://sourceforge.net/p/sevenzip/discussion/45797/thread/73ed3ad7/ Thanks for the reply, Randall On Tue, Jan 21, 2014 at 2:19 PM, Randall Farmer <randall(a)wawd.com> wrote: > > For external uses like XML dumps integrating the compression > > strategy into LZMA would however be very attractive. This would also > > benefit other users of LZMA compression like HBase. > > For dumps or other uses, 7za -mx=3 / xz -3 is your best bet. > > That has a 4 MB buffer, compression ratios within 15-25% of > current 7zip (or histzip), and goes at 30MB/s on my box, > which is still 8x faster than the status quo (going by a 1GB > benchmark). > > Re: trying to get long-range matching into LZMA, first, I > couldn't confidently hack on liblzma. Second, Igor might > not want to do anything as niche-specific as this (but who > knows!). Third, even with a faster matching strategy, the > LZMA *format* seems to require some intricate stuff (range > coding) that be a blocker to getting the ideal speeds > (honestly not sure). > > In any case, I left a note on the 7-Zip boards as folks have > suggested: https://sourceforge.net/p/sevenzip/discussion/45797/thread/73ed3ad7/ > > Thanks for the reply, > Randall > >

1 0

Compressing full-history dumps faster
by Randall Farmer 21 Jan '14

21 Jan '14

Hi, everyone. tl;dr: New tool compresses full-history XML at 100MB/s, not 4MB/s, with the same avg compression ratio as 7zip. Can anyone help me test more or experimentally deploy? As I understand, compressing full-history dumps for English Wikipedia and other big wikis takes a lot of resources: enwiki history is about 10TB unpacked, and 7zip only packs a few MB/s/core. Even with 32 cores, that's over a day of server time. There's been talk about ways to speed that up in the past.[1] It turns out that for history dumps in particular, you can compress many times faster if you do a first pass that just trims the long chunks of text that didn't change between revisions. A program called rzip[2] does this (and rzip's _very_ cool, but fatally for us it can't stream input or output). The general approach is sometimes called Bentley-McIlroy compression.[3] So I wrote something I'm calling histzip.[4] It compresses long repeated sections using a history buffer of a few MB. If you pipe history XML through histzip to bzip2, the whole process can go ~100 MB/s/core, so we're talking an hour or three to pack enwiki on a big box. While it compresses, it also self-tests by unpacking its output and comparing checksums against the original. I've done a couple test runs on last month's fullhist dumps without checksum errors or crashes. Last full run I did, the whole dump compressed to about 1% smaller than 7zip's output; the exact ratios varied file to file (I think it's relatively better at pages with many revisions) but were +/- 10% of 7zip's in general. Also, less exciting, but histzip's also a reasonably cheap way to get daily incr dumps about 30% smaller. Technical datadaump aside: *How could I get this more thoroughly tested, then maybe added to the dump process, perhaps with an eye to eventually replacing for 7zip as the alternate, non-bzip2 compressor?* Who do I talk to to get started? (I'd dealt with Ariel Glenn before, but haven't seen activity from Ariel lately, and in any case maybe playing with a new tool falls under Labs or some other heading than dumps devops.) Am I nuts to be even asking about this? Are there things that would definitely need to change for integration to be possible? Basically, I'm trying to get this from a tech demo to something with real-world utility. Best, Randall [1] Some past discussion/experiments are captured at http://www.mediawiki.org/wiki/Dbzip2, and some old scripts I wrote are at https://git.wikimedia.org/commit/operations%2Fdumps/11e9b23b4bc76bf3d89e1fb… [2] http://rzip.samba.org/ [3] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.11.8470&rep=rep1&t… [4] https://github.com/twotwotwo/histzip

3 4

Setting up a local Wikipedia mirror with extensions
by Bastian koell 17 Jan '14

17 Jan '14

Hi, I am trying to set up a local Wikipedia mirror. While I was reading up on how to import xml dumps and install extensions manually - I find it hard to match and install all the extensions manually and properly. I have been testing this with the Simple Wikipedia. What is the best and easiest way to install a local mirror? Best, Bastian

2 1

Preparing for wp-mirror-0.7
by wp mirror 11 Jan '14

11 Jan '14

Dear Ariel, Happy New Year. I am gearing up for wp-mirror-0.7. To that end, I would like to list some issues that I see; and I would like to offer my help in solving them. 0) Problem Statements 0.1) Page Rendering. Wp-mirror-0.6 works well in the sense that it builds a faithful mirror of any of your wikis. However, during 2013 the rendering of pages eroded materially. For example, o interlanguage links have vanished both from rendered pages and from dump files; o infoboxes are no longer rendered; o most transclusions now render as redlinks even though the templates are easily found in the underlying database; etc. I understand that this erosion occurred because wp-mirror-0.6 still uses mediawiki-1.19, whereas WMF has moved on to mediawiki-1.23. For example, I understand that: o interlanguage links have been removed to the wikidata project, the rendering of which requires mediawiki-1.21+; o infoboxes now require the scribunto extension which requires mediawiki-1.20+ 0.2) Database Schema. Some differences in database schema have appeared. o category - dump files now have 5 fields, whereas the database schema has 6 fields; o exterallinks - dump files now have 4 fields, whereas the database schema has 3 fields. Loading these two tables generate the error message: ``Column count doesn't match value at row 1.'' 0.3) Version Lifecycle. According to < http://www.mediawiki.org/wiki/Version_lifecycle> mediawiki 1.23 LTS is slated for May 2014. However, the Debian packaging team is silent as to their plans for a transition from mediawiki-1.19 LTS to mediawiki-1.23 LTS. 0.4) Image Dumps. The large image dump tarballs are now a year old. This means that, while wp-mirror still downloads the bulk of its images from these tarballs, there are a growing number that must be downloaded individually from WMF. 0.5) Thumbs. One person has asked me if dump files of thumbs could be made available. We are beginning to see thumb dumps from the xowa project. 0.6) IPv6. I am glad to see that <gerrit.wikimedia.org> has an IPv6 address. However, <bastion.wmflabs.org> still does not. My internal network is IPv6 only. 1) mwxml2sql This utility from Ariel Glenn has proved invaluable to the wp-mirror project. This utility, together with MySQL 5.5 fast index creation, allows wp-mirror to build mirrors much faster than before (80% less time). 1.1) Need for update. According to its version information, mwxml2sql may only be valid through mediawiki-1.21. (shell)$ mwxml2sql --version mwxml2sql 0.0.2 Supported input schema versions: 0.4 through 0.8. Supported output MediaWiki versions: 1.5 through 1.21. Whereas, I am looking forward to mediawiki-1.23 LTS (see below), I would like to know if mwxml2sql should be updated. 1.2) Help Offer. If mwxml2sql does need updating, I would be happy to help with this; and to package it for Debian as I have done before. Perhaps we could call it mwxml2sql-0.0.3. 2) mediawiki-1.23 LTS. 2.1) Vision. I would like wp-mirror-0.7 to be able to build a mirror that serves pages that look no different than those served by WMF. 2.2) DEB package. To that end, I am thinking of packaging mediawiki-1.23 together with the extensions needed for rendering WMF wikis with wikidata content, infoboxes, math, transclusions, etc. Given WMF's ``continuous integration'' development model, I would like to be able to automatically generate a tarball and DEB package each time WMF pushes an update to its servers. 2.3) Debian package repository. Such a DEB package would be distributed with wp-mirror. In preparation for this, I have set up a Debian package repository at <http://download.savannah.gnu.org/releases/wp-mirror/>. It is currently used to distribute wp-mirror-0.6 and an unstable version of wp-mirror-0.7. Home page <http://www.nongnu.org/wp-mirror/>. 2.4) Help Offer. I am happy to do most of this work myself. However, I will need some guidance on interacting with the appropriate GIT repositories. I hope that you can put me in touch with someone involved in the ``continuous integration'' process. 3) Media dumps I am thinking that updating the image dumps annually would be adequate. Including thumbs in those dumps would materially assist the off-line community. I could easily update wp-mirror-0.7 to give the user a choice (no media files, thumbs only, full size media files). Sincerely Yours, Kent

2 1

← Newer
1
...
38
39
40
41
42
43
44
...
72
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Xmldatadumps-l