Wikitech-l September 2010

wikitech-l@lists.wikimedia.org

102 participants
96 discussions

testing one phase at a time parallelized en wiki dumps
by Ariel T. Glenn 05 Sep '10

05 Sep '10

Hello folks, Today I shot the full history en wiki dumps that claimed they would take three months to complete. I started a new en wiki run which tests out two new features: running jobs step by step (in arbitrary order, assuming any dependencies have been run) and breaking up the xml file creation into chunks that run in parallel. Because of this, you may notice some funkiness with the status pages over the next little while; things like the progress line are going to be out of whack, and I am sure we will find new and exciting bugs (though hopefully not in the dump file output). We have some bizarre behavior around the index pages which all seem to claim that the given date is its own previous dump. I'll be looking into it, but in the meantime, the old dumps are available at: http://dumps.wikimedia.org/enwiki/20100817/ http://dumps.wikimedia.org/enwiki/20100730 http://dumps.wikimedia.org/enwiki/20100130 http://dumps.wikimedia.org/enwiki/20100116 http://dumps.wikimedia.org/enwiki/20091103 The parallelizing scheme just does dumps of n pages in sequence (where n is arbitrarily set at 2 million right now), including all revisions or not, depending on the dump. This shouldn't screw with anyone's code that depends on the page IDs to be in order. This is only being tested out on the en wiki dumps at present; all other jobs will run just as they used to. Ah, I wonder if anyone out there would be interested in working on dbzip2 or seeing if it is still needed; this is a parallelizing bzip2 with some features that pbzip2 doesn't have. See http://www.mediawiki.org/wiki/Dbzip2 This could potentially save us time in the recombine phase of the bzip2 history dumps, if people want to have those available as one file nad not just as separate pieces. We don't have even a start on that for 7zip, so that's another thing for someone to look into... any takers? Ariel Glenn

1 0

Re: [Wikitech-l] Community vs. centralized development
by Danese Cooper 04 Sep '10

04 Sep '10

Okay, so here's my take... First of all I want to thank Aryeh for taking the time to write out the original mail in the thread. You've obviously done quite a bit of thinking. I realize there has been discontent and even concern with the way things have been / are between Foundation tech staff / paid contractors and "everyone else" on the volunteer dev side of this Project upon which we all toil. Aryeh, your mail was a well articulated instance of those feelings. I'm not going to deconstruct your suggestions; I actually agree with several of them (although I think you'd find the reality of flagship open source projects somewhat different than the legend). Instead I'm going to make a more general statement about why and how we're trying to shape our future efforts... As I came into WMF, the UX project was still in full swing and we decided not to interrupt their momentum until the Stanton Grant was finished. I focused my efforts instead on planning for the new Fiscal Year's budget and hiring plans and on observing the work patterns of the existing Tech staff, the expectations and needs of the Programs staff and the community interaction dynamic. During most of this time I've been close to silent on public lists because my opinions were still in formation. I figured there was nothing worse than somebody rushing in with a lot of uninformed blathering. Gradually I decided these were the most serious problems (in order of importance) to tackle with a new design WMF Tech: 1. Eliminate single points of failure / bottlenecks 2. Reconfigure into teams designed to encourage faster (shorter duration) and more accurate projects / deployments 3. Develop programs to encourage / grow volunteers into paid staff...recognizing that as a non-profit WMF needs to plan for more frequent turnover in the Tech department to ensure that we can grow expertise across the dev community rather than concentrating it in a few core people. 4. Restore the balance between community-led and foundation-led efforts so WMF feels again like a partnership rather than concentric circles of permission / blame Once I recognized these objectives, I started working out how to reach them. Notice that they are in order of importance above...starting with hiring / training / documenting and otherwise creating redundancies to mitigate the SPOF problem (in Operations to keep the sites functioning for instance) is a big focus of the hiring plan for this Fiscal Year. Fully 1/3 of the Tech hires in this fiscal year are SPOF related. Item 2 is necessary to again make us very productive (as at least one respondent on the current thread believes we should be given there are "more of us than ever before"). Experienced developers know that more hands aren't necessarily better unless you can modularize work to form small agile teams within the larger structure to get discreet bits of work done. The Linux Kernel team does this, and Apache projects are implicitly organized this way. Most of the expense of this re-org hits in the next fiscal year (hiring engineering program managers to help teams stay undistracted and productive and to maintain an overview so discreet bits of work add up to the correct whole), so this year about 1/3 of the Tech hiring will serve this item. But the crux of Aryeh's original email is not about the first two items on my list...it really rests in items 3 and 4. It is my belief that the Foundation can not and should not plan to fund or achieve most development projects cathedral-style. To do so would not be consistent with our founding principles (and in any case wouldn't be possible given our resources). At the same time our Projects have grown in both complexity and popularity to the point where we can't we solely expect volunteerism to provide for all needs. There has to be a partnership...not a detente. To that end it is clear we need to improve transparency and afford more opportunities for participation and cooperation between paid Tech staff and the volunteer developer community. We need to undertake this shift with realistic expectations. We are never going to be interested in every scrap of volunteer engineering ever undertaken, but we are going to take steps to make it easier for volunteer engineers wishing to contribute to figure out what contributions are sought and how to gain increased responsibility in our essentially meritocratic community (which is how Mozilla finds their new paid staff as well). We're going to create more entry points and to get volunteers more involved. In this fiscal year 1/3 of the new Tech hires are in service of items 3 and 4... If you're still reading, then you must really care to know what I think. Thank you for that. Mostly I wanted to share that re-balancing the volunteer-to-paid staff dynamic is definitely a first-year priority for me, although its not more urgent than establishing a new primary Data Center and making sure we have staff to keep the site running...but its coming. We're going to co-create (or re-create it). Like all work of digital "intentional communities"[1], its going to take work on all sides to get it right...and I hope you'll help. Danese Cooper CTO, Wikimedia Foundation danese(a)wikimedia.org [1] http://en.wikipedia.org/wiki/Intentional_community

3 2

[HTML5] Improving Commons upload interface
by Christophe Henner 04 Sep '10

04 Sep '10

Hi Everyone, I searched the archives of both list and couldn't find any thread about it. I could miss it, so sorry it it already has been discussed. We all know that uploading a file on commons, and on Wikimedia, is kind of tricky nowdays. However, this could be changed thanks to HTML5. HTML5 includes the drag and drop thingy that makes the uploads easier and that can automatically fetch the EXIF datas. If we want we could even allow the multiple drag and drop. Anyway this could be a solution to look at, you can get more information there : http://hacks.mozilla.org/2009/12/firefox-36-fileapi-demo-reading-exif-data-… and there : http://hacks.mozilla.org/2009/12/file-drag-and-drop-in-firefox-3-6/ All the best, -- Christophe

16 34

Re: [Wikitech-l] latextwowki: latex to mediawiki translator
by Asia Jedrzejewska-Szmek 03 Sep '10

03 Sep '10

Dear Jason, I wrote the parser mostly beacuse I've got lot's of latex files to convert. I have used some of the converters I found on the web -- e.g. latex2wiki, but they either were insufficient or hanged on the files I had. I have just tried pandoc and it is very nice, but it does not support e.g. newcommands, figures, tables, references and bibliography sufficiently. I've parsed one of my articles with pandoc and latextwowiki and you can see the difference: http://brain.fuw.edu.pl/edu/Test/parserart (transalted with latextwowiki) and http://brain.fuw.edu.pl/edu/Test/pandoc (translated with pandoc). Cheers, Asia

1 0

Re: [Wikitech-l] converter from LaTeX to MediaWiki
by Asia Jedrzejewska-Szmek 03 Sep '10

03 Sep '10

Dear Juan, Sorry for the bug, it should be fixed now. The problem with the REPLACEMENT might be due to two reasons: a) you should use Python 3 beacuse of simpler unicode support, b) there is something in your latex files, that I didn't expect writing my parser. I such case, please do send me your file (if it is possible) and I'll try to fix the parser. Argparse is/was an external module, but is going to be in the Python Standard Library from Python 3.2 on. Cheers, Asia

1 0

latextwowki: latex to mediawiki translator
by asia Jędrzejewska-Szmek 01 Sep '10

01 Sep '10

I would like to announce latextwowiki -- a latex to mediawiki translator. It uses tralics to transform latex to xml. Latextwowiki is written in Python3.1 and uses lxml. Code is under GPL v.3 or later. The pre-alpha version can be downloaded from http://escher.fuw.edu.pl/git/latextwowiki. Features: * translates \ref{} to sections to proper mediawiki references in [[]] * various complicated latex equations work correctly * supports equation, figure, table references via Extension:CrossReference * latex bibliography support via Extension:Cite. Every comment, bug and hairy latex file to test will make me happy. Asia Jędrzejewska-Szmek

2 1

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l September 2010