Xmldatadumps-l

xmldatadumps-l@lists.wikimedia.org

1 participants
731 discussions

Request for input on de fr wiki breaking change to dumps!
by Ariel T. Glenn 02 Dec '15

02 Dec '15

I assume you have all seen https://phabricator.wikimedia.org/T116907 "Explore the possibility of splitting dewiki and frwiki into smaller chunks" If not, and you ever use frwiki or dewiki page content dumps, go read it now. Or if you know of anyone who uses them, please nag them to go read it. The upshot is that we will most likely on January 1st 2016 do all further dump runs of frwiki and dewiki with so-called 'checkpointing'. This change is being made so that if one of these jobs is interrupted for whatever reason, it can be rerun with only the missing page ranges dumped on the second run, saving quite a lot of time. A second reason is to ease the burden on downloaders, who generally prefer downloading several smaller files rather than one large 90gb file (example taken from dewiki history dumps). WHat does this mean in practice for you, users of the dumps? It means that filenames for the page content (article, meta-current and meta -history) dumps will have pXXXXpYYYY in the names, where XXXX is the first page id in the file and YYY is the last pageid in the file. For examples of this you can look at the enwiki page content dumps, which have been running that way for a few years now. This notice should give you plenty of time to convert your tools to use the new nameing scheme. I encourage you to forward this message to other appropriate people or groups. Thanks, Ariel

2 3

Dumps 2.0 Wiki Dev Summit but also...
by Ariel T. Glenn 01 Dec '15

01 Dec '15

Also if you are a dumps user or have thoughts about how you would redo them from scratch, get your ideas in now. We're not waiting for the Dev Summit to get the work started. See https://phabricator.wikimedia.org/T114019 for details, especially the document linked at the end of the task description under 'FOR CURRENT DISCUSSION'. We need: comments on the strawman proposed model in the document mentioned above; proposals for code we can reuse for any of those pieces, especially the job queue/management piece. (Celery? Something else?) What should the object store be based on, if we have one? Is Ceph a dead end? Will Swift be deadly slow or should we use it since we have it in house already? Etc. All comments on the ticket please so we have them all in one place. Expect that most of the code written here at WMF will be in python unless someone else volunteers to write some. Anyone interested in doing some of the development can step right up too of course. Please forward this on to other fora as appropriate. Ariel

1 0

Redirect dump format
by Alan Said 16 Nov '15

16 Nov '15

Hi all, I want to recreate the redirect graph form the redirect sql file. Having loaded the file into a local MySQL database I have a table with the fields listed in https://www.mediawiki.org/wiki/Manual:Redirect_table (except the rf_interwiki column). For instance, the second row in the table contains the values: rd_from rd_namespace rd_title 13 0 History_of_Afghanistan I interpret this as page ID 13 in namespace 0 redirects to https://en.wikipedia.org/wiki/History_of_Afghanistan Now, trying to use the id in the rf_from column, I attempt to access the page information using the example at https://www.mediawiki.org/wiki/Manual:Page_table#page_id i.e. loading the url https://www.mediawiki.org/w/api.php?action=query&prop=info&pageids=13 The result tells me that pageid 13 is missing. So, can any one please tell me what I'm doing wrong? Best, Alan -- Alan Said Recorded Future e: alansaid(a)acm.org t: @alansaid w: www.alansaid.com

2 1

Re: [Xmldatadumps-l] [Wiki-research-l] Download of pageviews dataset
by Federico Leva (Nemo) 11 Nov '15

11 Nov '15

Cristian Consonni, 11/11/2015 15:09: > I am working with a student on scientific citation on Wikipedia and, > very simply put, we would like to use the pageview dataset to have a > rough measure of how many times a paper was viewed thanks to > Wikipedia.[*] > > The full dataset is, as of now, ~ 4.7TB in size. > > I have two questions: > * if we download this dataset this would entail, from a first > estimation, ~ 30 days of continuous download (assuming an average > download speed of ~ 2MB/s, which was what we measured over the > download of a month of data (~ 64GB)). Here at my University (Trento, > Italy) this kind of downloads have to be notified to the IT > department. I was wondering if this would be a useful information for > the WMF, too. No need to notify such small downloads. > * given the estimation above I was wondering if it is possible to > obtain this data over FedEx Bandwith (cit. [1]). i.e. via shipping of > a physical disk, I know that in some fields (e.g. neuroscience) this > is the standard way to exchange big dataset (in the order of TBs). This assumes that some point of the network has faster download from that machine. The server is very slow for pretty much anyone except rare exceptions (https://phabricator.wikimedia.org/T45647 ), possibly even inside the cluster. Copying to a hard drive might take many days. You have two more alternatives: * scp from Labs, /public/dumps/pagecounts-all-sites/ (sometimes reaches 3-4 MB/s for me); * archive.org for pagecounts-raw https://archive.org/search.php?query=wikipedia_visitor_stats (you can start download of all months at once and use torrent, will hopefully saturate your bandwidth because you will download from dozens servers rather than one). Nemo > > Thanks in advance for your help. > > Cristian > [*] I know these are pageviews and not unique visitors, furthermore > there is no guarantee that viewing a citation means anything. I am > approaching to this data the same way "impressions" versus > "clicktroughs" are treated in the online advertising world. > [1] https://what-if.xkcd.com/31/ > > _______________________________________________ > Wiki-research-l mailing list > Wiki-research-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l >

1 0

Proposal: Splitting of dewiki and frwiki into smaller chunks
by Hydriz Scholz 10 Nov '15

10 Nov '15

Hi all, If you are a user of the dewiki and frwiki dumps, please provide your input on the proposal for splitting the dewiki and frwiki dumps into smaller pieces, similar to that of enwiki. The proposal is available on Phabricator. [1] All comments are welcome! Also, as a reminder, please provide your comments for the next generation of dumps on Phabricator. [2] This is your chance to propose changes to the dumps to suit your needs. Thanks! [1]: https://phabricator.wikimedia.org/T116907 [2]: https://phabricator.wikimedia.org/T114019 -- Best regards, Hydriz Scholz

1 0

Wikimedia Dev Summit 2016 dumps 2.0 redesign session
by Ariel T. Glenn 30 Oct '15

30 Oct '15

Planning for the full redesign to kick off early 2016. When I say "full redesign" I mean it, so let's not talk small stuff about whether json or xml is a batter format; let's talk about a design that will allow us to plug in various formats easily, for example. Starting point for discussion here: https://phabricator.wikimedia.org/T114019 See the last link on the description. What we can start on now: are these the right components to break the dumps into? Can we indeed cover all sorts of dumps this way? What existing software can we re-use? WHat sort of grid computing or other package can we use for the "black box" in the diagram? Etc. etc. We want to get as much of this hashed out ahead of time so that we can get the maximum out of the session. Ariel

3 2

Still no dumps available.
by Bryan White 20 Oct '15

20 Oct '15

Dumps are still not available on labs after two weeks. No one has said anything on why they are down or if they will be fixed. Dumps are still not being run twice a month. Bryan

2 1

Non-English dump problems.
by Chris Newell 09 Oct '15

09 Oct '15

Hi, Although the English language dump for October has completed successfully there are problems with some other languages - the Spanish, German and Russian dumps dated 20151002 failed at: "Articles, templates, media/file descriptions, and primary meta-pages". Could someone could look into this? Thanks, Chris ------------------- Chris Newell Lead Technologist Internet Research & Future Services BBC Research & Development

2 1

Incremental dump generation is stopped
by Ivan 02 Oct '15

02 Oct '15

Hello, Daily dump generator did not create any dumps during last two days. Directories: http://dumps.wikimedia.org/other/incr/wikidatawiki/20150930/ http://dumps.wikimedia.org/other/incr/wikidatawiki/20151001/ do not contain any dumps. Is it known issue? Sincerely, Ivan

5 4

Use of dumps in mediawiki
by Yoni Lamri 07 Sep '15

07 Sep '15

Hello everybody, First thread in the list, i'm relatively new to wikipedia/media usage (1 month). I followed the doc (RTFM as usual) and every steps seems to finish in a wall. My simple question, how to correctly install a wikipedia mirror from dumps in mediawiki ? My goal: Create an offline wikimedia server, from FR, EN or PT dumps, (1 language only). I did some tests with the wowiki which is small enough for testing purpose but I encountered problems: The tools mentioned in the doc are 404 for most of them or outdated. for example the xml2sql or mwdumper. the original importer of wikimedia takes 20 minutes to import the wowiki 1.6Mo dump which means approximatly a decade for the FR one... I tried with the last MWDumper found on github, it can quickly generate a good sql file but it seems that special characters in page text are not "slashed" so \n return SQL error... Some columns are missing in original mediawiki installer, the "page_counter" in "page" table for example, is there some needed extensions to install dumps? How can i found/get medias (images low res are enough) How to do with interwiki medias/links/pages? To follow next step will be updates procedure, incremental or complete... Seems french wikimedia team are stuck with my questions.. I'm not closed to development (python, PHP or JEE) but i need an entry point to begin if nobody work on it... Thank you for your work, i know you are quite busy with the dump generation but our project is quite serious and lots of people want it and we need quick answers for the feasability. Best regards, Yoni

4 5

← Newer
1
...
30
31
32
33
34
35
36
...
74
Older →

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Xmldatadumps-l