Xmldatadumps-l

xmldatadumps-l@lists.wikimedia.org

720 discussions

Help for dump
by Yannick Guigui 02 Jun '14

02 Jun '14

Hi to all, I have a problem and I need your help. It's about Dump of Wikipedia. I'm working in a project which uses Wikipedia Database to display articles offline. I need your help in two points: - - First, the problem about the database in Ubuntu is that in my Country we have several interruptions of electricity and it's very difficult for us to maintain a quite installation with one interruption of electricity every hour. We use to start again the installation after purge the some table in the data base each time. - 7 The second problem is that we have not a good bandwish of internet to download dump of image of Wikipedia because it's too slow. After try to resolve these problems, our evolution is still very slow. Please we have a friend in USA who back in few weeks. Please can we send you two hard discs to you for help us to install a local copy of Wikipedia in French and English with image in articles in mediawiki? Please this solution is the best for our environment and situation in this time. Thank's for your support and assistance to these dumps for developer and for the idea to make information accessible for all and open. Tank's (sorry for english)

2 1

Fwd: fast dump importer and revision diff calculator
by wp mirror 01 Jun '14

01 Jun '14

---------- Forwarded message ---------- From: wp mirror <wpmirrordev(a)gmail.com> Date: Sun, Jun 1, 2014 at 4:36 PM Subject: Re: [Xmldatadumps-l] fast dump importer and revision diff calculator To: Jeremy Baron <jeremy(a)tuxmachine.com> Dear Jeremy, I am interested. But I have a few questions: o How much RAM is `copious'? o Do you have any test results? o Can it be used with `pages-articles' and `pages-meta-current'; or only with `pages-meta-history'? Sincerely Yours, Kent On Fri, May 30, 2014 at 6:31 AM, Jeremy Baron <jeremy(a)tuxmachine.com> wrote: > fast dump importer and revision diff calculator. > > I believe it's really fast but needs copious RAM. > > https://github.com/makoshark/wikiq/blob/master/README > > -Jeremy > > _______________________________________________ > Xmldatadumps-l mailing list > Xmldatadumps-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l > >

1 0

fast dump importer and revision diff calculator
by Jeremy Baron 30 May '14

30 May '14

fast dump importer and revision diff calculator. I believe it's really fast but needs copious RAM. https://github.com/makoshark/wikiq/blob/master/README -Jeremy

1 0

mirror connectivity
by wp mirror 28 May '14

28 May '14

Dear Kevin, It looks like <ftpmirror.your.org> may be down. 0) Time. I am not sure when it happened. Probably between 01:00 and 09:00 EDT. 1) TCP level. RSYNC, HTTP, and FTP all appear to be down. (shell)$ rsync ftpmirror.your.org:: rsync: failed to connect to ftpmirror.your.org(2001:4978:1:420::cc09:3752): Connection timed out (110) rsync: failed to connect to ftpmirror.your.org (204.9.55.82): Network is unreachable (101) rsync error: error in socket IO (code 10) at clientserver.c(128) [Receiver=3.1.0] (shell)$ wget http://ftpmirror.your.org/ --2014-05-28 13:07:32-- http://ftpmirror.your.org/ Resolving ftpmirror.your.org (ftpmirror.your.org)... 2001:4978:1:420::cc09:3752, 204.9.55.82 Connecting to ftpmirror.your.org (ftpmirror.your.org)|2001:4978:1:420::cc09:3752|:80... failed: Connection timed out. Connecting to ftpmirror.your.org (ftpmirror.your.org)|204.9.55.82|:80... failed: Network is unreachable. (shell)$ ftp ftpmirror.your.org ftp: connect: Connection timed out ftp> quit 2) IP level. I can reach <ftpmirror.your.org> with PING but not with PING6. (shell) ping -c 1 ftpmirror.your.org PING ftpmirror.your.org (204.9.55.82) 56(84) bytes of data. 64 bytes from ftpmirror.your.org (204.9.55.82): icmp_req=1 ttl=52 time=33.9 ms --- ftpmirror.your.org ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 33.958/33.958/33.958/0.000 ms (shell)$ ping6 -c 1 ftpmirror.your.org PING ftpmirror.your.org(ftpmirror.your.org) 56 data bytes --- ftpmirror.your.org ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms 3) Announcement. By the way, when you take a site down for maintenance, is there anyplace I can look to see if there was an announcement? Sincerely Yours, Kent

2 1

Dump file for `sites' table
by wp mirror 13 May '14

13 May '14

Dear Ariel, I have a questions about the dump file for the `sites' table. 1) Dump file exists, and contains meaningful data. (shell)$ rsync ftpmirror.your.org::wikimedia-dumps/wikidatawiki/20140508/ | grep sites -rw-rw-r-- 15,817 2014/05/11 00:15:46 wikidatawiki-20140508-sites.sql.gz (shell)$ rsync --progress ftpmirror.your.org::wikimedia-dumps/wikidatawiki/20140508/wikidatawiki-20140508-sites.sql.gz . wikidatawiki-20140508-sites.sql.gz 15,817 100% 15.08MB/s 0:00:00 (xfr#1, to-chk=0/1) (shell)$ zless wikidatawiki-20140508-sites.sql.gz 2) It is mentioned in `dumpruninfo.txt' (shell)$ rsync --progress ftpmirror.your.org::wikimedia-dumps/wikidatawiki/20140508/dumpruninfo.txt . dumpruninfo.txt 2,293 100% 2.19MB/s 0:00:00 (xfr#1, to-chk=0/1) (shell)$ cat dumpruninfo.txt | grep sitesname:sitestable; status:done; updated:2014-05-11 04:15:48 name:sitestatstable; status:done; updated:2014-05-08 17:39:36 3) But it is not mentioned in `md5sums.txt' (shell)$ rsync --progress ftpmirror.your.org::wikimedia-dumps/wikidatawiki/20140508/wikidatawiki-20140508-md5sums.txt . wikidatawiki-20140508-md5sums.txt 2,189 100% 2.09MB/s 0:00:00 (xfr#1, to-chk=0/1) (shell)$ cat wikidatawiki-20140508-md5sums.txt | sites bash: sites: command not found Sincerely Yours, Kent

2 3

mirror connectivity
by wp mirror 01 May '14

01 May '14

Dear Kevin, Early on 2014-May-01, just past midnight (EDT), a download using `wget' broke off. Since then, I cannot establish an HTTP connection to < http://dumps.wikimedia.your.org>. I can establish an RSYNC connection to get the list of modules: (shell)$ rsync ftpmirror.your.org:: FreeBSD FreeBSD-Archive FreeBSD-CVS wikimedia-dumps wikimedia-images wikimedia-imagedumps centos NetBSD pkgsrc kiwix everything Early this morning, I was able to list the contents of the `kiwix' module and part of the contents of the `everything' module, but could not list the contents of any of the `wikimedia-*' modules. This evening, I can not list the contents of any of the modules: (shell)$ rsync ftpmirror.your.org::wikimedia-dumps/ @ERROR: max connections (16) reached -- try again later rsync error: error starting client-server protocol (code 5) at main.c(1534) [Receiver=3.0.9] Sincerely Yours, Kent

2 1

Hindi
by Benoit Lelong 24 Apr '14

24 Apr '14

Hi all, Has anybody already tried to compute the hindi wikipedia dump ? If yes, or if somebody intend to do it I am ready for some collaboration. All the best, Benoit.

2 1

Re: [Xmldatadumps-l] [Wikitech-l] Compressing full-history dumps faster
by Randall Farmer 24 Apr '14

24 Apr '14

Ack, sorry for the (no subject); again in the right thread: > For external uses like XML dumps integrating the compression > strategy into LZMA would however be very attractive. This would also > benefit other users of LZMA compression like HBase. For dumps or other uses, 7za -mx=3 / xz -3 is your best bet. That has a 4 MB buffer, compression ratios within 15-25% of current 7zip (or histzip), and goes at 30MB/s on my box, which is still 8x faster than the status quo (going by a 1GB benchmark). Trying to get quick-and-dirty long-range matching into LZMA isn't feasible for me personally and there may be inherent technical difficulties. Still, I left a note on the 7-Zip boards as folks suggested; feel free to add anything there: https://sourceforge.net/p/sevenzip/discussion/45797/thread/73ed3ad7/ Thanks for the reply, Randall

2 3

Tool for incremental dumps extraction
by Christophe Desclaux 18 Apr '14

18 Apr '14

Hello, I would like to have a French wikipedia mirror updated daily, so i looked at incremental dumps on <http://dumps.wikimedia.org/other/incr/frwiki/> and i think it's what i need. But I don't found tools in order to extract this dumps. The mwimport script only worked on frwiki-20140417-pages-meta-hist-incr.xml and doesn't work on the stubs file. Is there a tool working specifically for the incremental dumps? I looked the GSoC 2013 about this functionality (and spent a lot of time on this mailing list archive): On <https://www.mediawiki.org/wiki/User:Svick/Incremental_dumps> there is no informations about an extracting tool. <https://gerrit.wikimedia.org/r/#/projects/operations/dumps/incremental,dash…> is empty :(. Nothing also on the student's github <https://github.com/svick> On <http://www.mediawiki.org/wiki/Incremental_dumps/File_format#Library> he is saying informations about the library but no more. Did I miss something? thanks, Christophe

1 0

Re: [Xmldatadumps-l] Updating mwxml2sql-0.0.2 for mediawiki 1.23 LTS
by wp mirror 10 Apr '14

10 Apr '14

Dear Ariel, Thank you for your guidance. I pushed another change to gerrit for review that should address the issue of the new `page_links_updated' field. Sincerely Yours, Kent On 2/7/14, Ariel T. Glenn <ariel(a)wikimedia.org> wrote: > Last reply: I double-checked the content/format model stuff, and the > only nagging question I have remaining is how well it works with > non-text handlers. But that would not be a new issue, and the code for > the base case is certainly correct. So I think we are down to just the > page_links_updated variable for > 1.22 and that would do it. > > Ariel > > Στις 05-02-2014, ημέρα Τετ, και ώρα 02:43 -0500, ο/η wp mirror έγραψε: >> Dear Ariel, >> >> I have been reading your code for `mwxml2sql-0.0.2' with a view >> towards updating it for mediawiki-1.23 LTS. >> >> 0) Support status >> >> Currently, the version info for `mwxml2sql' states the following: >> >> (shell)$ mwxml2sql --version >> mwxml2sql 0.0.2 >> Supported input schema versions: 0.4 through 0.8. >> Supported output MediaWiki versions: 1.5 through 1.21. >> >> 1) Current input schema version >> >> Currently, your XML dump files have the following header: >> >> (shell)$ head -n 1 zuwiki-20140121-pages-articles.xml >> <mediawiki xmlns="http://www.mediawiki.org/xml/export-0.8/" >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >> xsi:schemaLocation="http://www.mediawiki.org/xml/export-0.8/ >> http://www.mediawiki.org/xml/export-0.8.xsd" version="0.8" >> xml:lang="zu"> >> >> From this I gather that XML schema is still 0.8, and that `mwxml2sql' >> needs no update on that head. >> >> 2) Current output MediaWiki version >> >> I reviewed the database schema for the `page', `revision', and `text' >> tables: >> >> <https://www.mediawiki.org/wiki/Manual:Page_table>, >> <https://www.mediawiki.org/wiki/Manual:Revision_table>, and >> <https://www.mediawiki.org/wiki/Manual:Text_table> >> >> It appears that the most recent changes to the schema for these three >> tables occurred for mediawiki versions 1.21, 1.21, and 1.19, >> respectively. >> >> From this I gather that the database schema used for mediawiki 1.23 >> LTS is the same as that used for mediawiki 1.21; and therefore >> `mwxml2sql' needs no update on that head. >> >> 3) Recommended updates >> >> From a review of your code, I concluded that two minor changes would be >> useful. >> >> 3.1) mwxml2sql.c >> >> The following three lines: >> >> (shell)$ grep 21 mwxml2sql.c >> fprintf(stderr,"Supported output MediaWiki versions: 1.5 through >> 1.21.\n\n"); >> /* we know MW 1.5 through MW 1.21 even though there is no MW 1.21 yet >> */ >> if (mwv->major != 1 || mwv->minor < 5 || mwv->minor > 21) { >> >> should read >> >> fprintf(stderr,"Supported output MediaWiki versions: 1.5 through >> 1.23.\n\n"); >> /* we know MW 1.5 through MW 1.23 */ >> if (mwv->major != 1 || mwv->minor < 5 || mwv->minor > 23) { >> >> 3.2) mwxmlelts.c >> >> The following line: >> >> (shell)$ grep 21 mwxmlelts.c >> <generator>MediaWiki 1.21wmf6</generator> >> >> should read >> >> <generator>MediaWiki 1.23wmf10</generator> >> >> 4) Request >> >> Please let me know if you agree with the above assessment. If you do, >> I would be happy to submit the changes to >> <https://gerrit.wikimedia.org/> for review. >> >> Sincerely Yours, >> Kent > > >

1 2

← Newer
1
...
36
37
38
39
40
41
42
...
72
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Xmldatadumps-l