Wikitech-l September 2010

wikitech-l@lists.wikimedia.org

102 participants
93 discussions

Community vs. centralized development
by Jamie Morken 06 Sep '10

06 Sep '10

> > So it sounds like > respect is also centralized in the wikimedia foundation, please > include > me in your email to your underlings Tim, as I would also like to > have > respect, maybe it will mean my request for image dumps is taken > seriously!? It would be nice if respect was earned, but it > sounds like > Tim can create/enforce respect in a more simple way, sounds like > a good > leadership path to me.. NOT! :) > > > Has it occurred to you that maybe the one person in charge of both > dumps and media storage is a busy person who also happens to be on > vacation right now? Hi, No, but you bring up a good point, having a single person in charge of both of these departments is not working very well, since the "department of image dumps" hasn't actually released any publicly available image dumps in 4+ years, and the "department of xml dumps" has only released a single successful* full history dump of enwiki in 3+ years. *only partially successful as some of the revision history is missing from the dump cheers, Jamie > > Roan Kattouw (Catrope) >

1 0

Separate DB or shared table for GlobalTemplateLinks?
by Peter17 06 Sep '10

06 Sep '10

Hi everyone! My Google Summer of Code is finished, but the code I have written for the interwiki transclusion is not yet ready to be merged in trunk. Some optimizations have to be made and some little issues are still to be discussed. The main remaining question is: should I use a separate DB or a shared table to store the GlobalTemplateLinks table? [1] Following the advice of my mentor, Catrope, my code is currently using a separate DB (which name is stored in $wgGlobalDatabase) since it is the mechanism used by CentralAuth and it is very simple for users to set it up. Platonides thinks that I should be using a shared table, since my code is not an extension. However, a disadvantage of shared table is that currently, adding a shared table in $wgSharedTables will force the user to share the users table, which is not always wanted. I have further developments to accomplish on my code concerning database tables (such as creating a GlobalNamespaces table to store the distant namespaces names). I would like to solve this issue first, so that I don't have to rewrite everything later... Can you please provide your opinion about this? Thanks in advance Best regards [1] http://www.mediawiki.org/wiki/User:Peter17/Reasonably_efficient_interwiki_t… -- Peter Potrowl http://www.mediawiki.org/wiki/User:Peter17

4 3

Community vs. centralized development
by Jamie Morken 06 Sep '10

06 Sep '10

Hi, > > > I can say that despite being a nobody at Mozilla and having gotten > > only one (rather trivial) patch accepted, I feel like I'm > taken more > > seriously by most of their paid developers than by most of > ours. > > I'm sorry to hear that, and I'd like to know (off list) which paid > developers are making you feel that way. I've made it pretty > clear to > Danese that I think you're one of our best volunteer developers, and > that you should be given whatever you ask for, which, I think it's > understood, includes respect. Maybe I can help to get that > message to > filter down to the rest of the organisation. > So it sounds like respect is also centralized in the wikimedia foundation, please include me in your email to your underlings Tim, as I would also like to have respect, maybe it will mean my request for image dumps is taken seriously! It would be nice if respect was earned, but it sounds like Tim can create/enforce respect in a more simple way, sounds like a good leadership path to me.. NOT! :) cheers, Jamie

2 1

Transaction nesting and savepoints
by Platonides 06 Sep '10

06 Sep '10

There are some places in our code where we are nesting transactions. A code like: $dbw->begin(); $dbw->insert( 'image', ... ); $article->doEdit( $pageText, $comment, EDIT_NEW | EDIT_SUPPRESS_RC ); $dbw->query( "UPDATE $site_stats SET ss_images=ss_images+1" ); $dbw->commit(); Is apparently safely grouped into a transaction, but the UPDATE won't be in the same transaction as the insert, since Article::doEdit will itself open a transaction, breaking the old one. This is an unexpected result, and hard to realise, should the second transaction lie several calls down (or even inside a hook!). The transaction break, common to all our supported databases is not even consistent among them: * Mysql will commit the open transaction if you begin another one [1]. * Postgres will ignore the second begin (and issue a warning) if you are inside a transaction [2]. * In sqlite an attempt to invoke the BEGIN command within a transaction will fail with an error [3] (our driver forces a commit there, matching mysql behavior). * Oracle is not using a specific statement in our driver begin(). Calling begin() inside another transaction will be silently ignored (matches postgres). * Mssql only takes into account the outermost transaction [4] (matches postgres). * Ibm_db2 driver is changing the autocommit property like Oracle, thus matching postgres. I propose changing Database::begin() / commit() / rollback() to keep the count in mTrxLevel and perform a savepoint instead of a BEGIN should it be called inside another one. Savepoints are a way to perform partial rollbacks inside a bigger transaction. They are supported by all our databases [5] [6] [7] [8] [9] [10]. We shouldn't even have problems with our 4.0.30 servers, since its support was added in mysql 4.0.14 [11]. 1-http://dev.mysql.com/doc/refman/5.1/en/implicit-commit.html 2-http://www.postgresql.org/docs/9.0/static/sql-begin.html 3-http://sqlite.org/lang_transaction.html 4-http://msdn.microsoft.com/en-us/library/ms188929.aspx 5-http://dev.mysql.com/doc/refman/5.1/en/savepoint.html 6-http://www.postgresql.org/docs/9.0/static/sql-savepoint.html 7-http://sqlite.org/lang_savepoint.html 8-http://www.dba-oracle.com/t_savepoint.htm 9-http://msdn.microsoft.com/en-us/library/ms378414.aspx 10-http://publib.boulder.ibm.com/infocenter/db2luw/v9/topic/com.ibm.db2.udb… 11-http://dev.mysql.com/doc/refman/4.1/en/savepoint.html

4 10

Bugzilla Weekly Report
by reporter 05 Sep '10

05 Sep '10

MediaWiki Bugzilla Report for August 30, 2010 - September 06, 2010 Status changes this week Bugs NEW : 100 Bugs ASSIGNED : 5 Bugs REOPENED : 21 Bugs RESOLVED : 90 Total bugs still open: 4889 Resolutions for the week: Bugs marked FIXED : 61 Bugs marked REMIND : 0 Bugs marked INVALID : 7 Bugs marked DUPLICATE : 12 Bugs marked WONTFIX : 4 Bugs marked WORKSFORME : 4 Bugs marked LATER : 3 Bugs marked MOVED : 0 Specific Product/Component Resolutions & User Metrics New Bugs Per Component LiquidThreads 11 Resource Loader 7 Site requests 4 Internationalization 3 General/Unknown 3 New Bugs Per Product MediaWiki 26 Wikimedia 14 MediaWiki extensions 19 Top 5 Bug Resolvers jeluf [AT] gmx.de 17 innocentkiller [AT] gmail.com 11 markus [AT] semantic-mediawiki.org 6 Platonides [AT] gmail.com 5 conrad.irwin [AT] gmail.com 4

1 0

Community vs. centralized development
by Jamie Morken 05 Sep '10

05 Sep '10

Hi, I think it would be a nice gesture if the wikimedia foundation decentralized some of the internal projects that have had little success over the last few years. Two that come to mind are the enwiki pages-meta-history file creation (1 successful dump in ~3 years), and apparently very little internal concern over this, and also the images/thumbnail wiki-specific dumps that have been neglected for several years. If these two projects are opened up to the community I think that would be great! cheers, Jamie

2 1

InlineEditor extension
by Jan Paul Posma 05 Sep '10

05 Sep '10

Whoops, I meant to address this mail to wikitech-l, not wikitext-l. ;-) Regards, Jan Paul > Hello, > > In commit 72458 I've added the InlineEditor extension. [1] This extension is a working implementation of the prototype(s) earlier posted on this list. It's not actually for use on live wikis, but more a proof-of-concept and framework to experiment with. I will explain the extension in detail for those of you who might be interested. > > == Design overview == > The extension exists of several parts, structured in sub-directories like the UsabilityInitiative extension. The InlineEditor extension itself provides a framework for different edit modes to build on. It displays the edit modes, provides an interface to mark editable pieces of wikitext, provides a client-side inline editor which the edit modes *may* use, is configurable with several fallback options to the full/traditional editor, and handles previewing, publishing, undo and redo. > > Every other extension provides an edit mode for the InlineEditor extension. They hook into InlineEditorMark and InlineEditorDefineEditors. The first one is called whenever wikitext is passed through the extension, and all edit modes can mark their editable pieces. Once this is done, a few algorithms will combine this with information of previously edited pieces, generate both wikitext to run through the parser, and JSON which is passed to the client, which maps the editable pieces to the original wikitext. The other hook is to include CSS, JS and messages to the page. > > == Limitations == > There are many things which are sub-optimal right now: > * The editor is slow. Whenever changing a small element and previewing it, the entire page is reparsed. This will be fixed by parsing only the element if possible (i.e. references have side effects at the bottom of the page). > * It's for now only possible to use the editor as primary editor, with a link to the full/traditional editor. There will be a configuration option whether to do this, or display a message at the top of the traditional edit page to switch to this editor. > * I've not tested things in older browsers (or IE at all, for that matter). I only know it runs fine in Firefox and Chrome, but it may have bugs in other browsers. > * The edit modes are really, really, basic right now. They may or may not screw things up. Most of them have just one or a few regular expressions which do well in general, but may fail at many edge-cases. > * The editor may not handle all the messages and edge cases of the traditional editor. > * The extensions is written for MediaWiki 1.16 but may or may not work with other versions. > > Also, I'm not sure at all whether the current set of edit modes is the way to go. Currently, they are mutually exclusive. Meaning that text marked by one editor is never included in text marked by another editor. However, maybe it's better to not have edit modes like this, but different granularity of editing. I.e. sentence => paragraph => block. This way the user will get familiar with more wikitext instead of always seeing small portions. The framework currently doesn't allow for overlap in markings, but I will work to make this possible. > > == Goals == > Goal of this extension is to provide a framework to easily play with different modes of editing in-line. Feel free to write extensions that use this framework, or help with the framework itself. Any usability or technical suggestions are also welcome! > > I hope to get some documentation up on mediawiki.org anytime soon, but note that the code is heavily documented inline. Feel free to ask any questions: I'm probably forgetting to mention some things that may not be clear to everyone. Also, there is no public wiki at the moment to test this extension with, will work on that, but if someone else can enable it on a test wiki that would be great too! > > To install the extension(s), check the instructions in /trunk/extensions/InlineEditor/InlineEditor.php. Thanks for your time reading this! > > Regards, > Jan Paul > > [1] http://www.mediawiki.org/wiki/Special:Code/MediaWiki/72458

1 0

testing one phase at a time parallelized en wiki dumps
by Ariel T. Glenn 05 Sep '10

05 Sep '10

Hello folks, Today I shot the full history en wiki dumps that claimed they would take three months to complete. I started a new en wiki run which tests out two new features: running jobs step by step (in arbitrary order, assuming any dependencies have been run) and breaking up the xml file creation into chunks that run in parallel. Because of this, you may notice some funkiness with the status pages over the next little while; things like the progress line are going to be out of whack, and I am sure we will find new and exciting bugs (though hopefully not in the dump file output). We have some bizarre behavior around the index pages which all seem to claim that the given date is its own previous dump. I'll be looking into it, but in the meantime, the old dumps are available at: http://dumps.wikimedia.org/enwiki/20100817/ http://dumps.wikimedia.org/enwiki/20100730 http://dumps.wikimedia.org/enwiki/20100130 http://dumps.wikimedia.org/enwiki/20100116 http://dumps.wikimedia.org/enwiki/20091103 The parallelizing scheme just does dumps of n pages in sequence (where n is arbitrarily set at 2 million right now), including all revisions or not, depending on the dump. This shouldn't screw with anyone's code that depends on the page IDs to be in order. This is only being tested out on the en wiki dumps at present; all other jobs will run just as they used to. Ah, I wonder if anyone out there would be interested in working on dbzip2 or seeing if it is still needed; this is a parallelizing bzip2 with some features that pbzip2 doesn't have. See http://www.mediawiki.org/wiki/Dbzip2 This could potentially save us time in the recombine phase of the bzip2 history dumps, if people want to have those available as one file nad not just as separate pieces. We don't have even a start on that for 7zip, so that's another thing for someone to look into... any takers? Ariel Glenn

1 0

Re: [Wikitech-l] Community vs. centralized development
by Danese Cooper 04 Sep '10

04 Sep '10

Okay, so here's my take... First of all I want to thank Aryeh for taking the time to write out the original mail in the thread. You've obviously done quite a bit of thinking. I realize there has been discontent and even concern with the way things have been / are between Foundation tech staff / paid contractors and "everyone else" on the volunteer dev side of this Project upon which we all toil. Aryeh, your mail was a well articulated instance of those feelings. I'm not going to deconstruct your suggestions; I actually agree with several of them (although I think you'd find the reality of flagship open source projects somewhat different than the legend). Instead I'm going to make a more general statement about why and how we're trying to shape our future efforts... As I came into WMF, the UX project was still in full swing and we decided not to interrupt their momentum until the Stanton Grant was finished. I focused my efforts instead on planning for the new Fiscal Year's budget and hiring plans and on observing the work patterns of the existing Tech staff, the expectations and needs of the Programs staff and the community interaction dynamic. During most of this time I've been close to silent on public lists because my opinions were still in formation. I figured there was nothing worse than somebody rushing in with a lot of uninformed blathering. Gradually I decided these were the most serious problems (in order of importance) to tackle with a new design WMF Tech: 1. Eliminate single points of failure / bottlenecks 2. Reconfigure into teams designed to encourage faster (shorter duration) and more accurate projects / deployments 3. Develop programs to encourage / grow volunteers into paid staff...recognizing that as a non-profit WMF needs to plan for more frequent turnover in the Tech department to ensure that we can grow expertise across the dev community rather than concentrating it in a few core people. 4. Restore the balance between community-led and foundation-led efforts so WMF feels again like a partnership rather than concentric circles of permission / blame Once I recognized these objectives, I started working out how to reach them. Notice that they are in order of importance above...starting with hiring / training / documenting and otherwise creating redundancies to mitigate the SPOF problem (in Operations to keep the sites functioning for instance) is a big focus of the hiring plan for this Fiscal Year. Fully 1/3 of the Tech hires in this fiscal year are SPOF related. Item 2 is necessary to again make us very productive (as at least one respondent on the current thread believes we should be given there are "more of us than ever before"). Experienced developers know that more hands aren't necessarily better unless you can modularize work to form small agile teams within the larger structure to get discreet bits of work done. The Linux Kernel team does this, and Apache projects are implicitly organized this way. Most of the expense of this re-org hits in the next fiscal year (hiring engineering program managers to help teams stay undistracted and productive and to maintain an overview so discreet bits of work add up to the correct whole), so this year about 1/3 of the Tech hiring will serve this item. But the crux of Aryeh's original email is not about the first two items on my list...it really rests in items 3 and 4. It is my belief that the Foundation can not and should not plan to fund or achieve most development projects cathedral-style. To do so would not be consistent with our founding principles (and in any case wouldn't be possible given our resources). At the same time our Projects have grown in both complexity and popularity to the point where we can't we solely expect volunteerism to provide for all needs. There has to be a partnership...not a detente. To that end it is clear we need to improve transparency and afford more opportunities for participation and cooperation between paid Tech staff and the volunteer developer community. We need to undertake this shift with realistic expectations. We are never going to be interested in every scrap of volunteer engineering ever undertaken, but we are going to take steps to make it easier for volunteer engineers wishing to contribute to figure out what contributions are sought and how to gain increased responsibility in our essentially meritocratic community (which is how Mozilla finds their new paid staff as well). We're going to create more entry points and to get volunteers more involved. In this fiscal year 1/3 of the new Tech hires are in service of items 3 and 4... If you're still reading, then you must really care to know what I think. Thank you for that. Mostly I wanted to share that re-balancing the volunteer-to-paid staff dynamic is definitely a first-year priority for me, although its not more urgent than establishing a new primary Data Center and making sure we have staff to keep the site running...but its coming. We're going to co-create (or re-create it). Like all work of digital "intentional communities"[1], its going to take work on all sides to get it right...and I hope you'll help. Danese Cooper CTO, Wikimedia Foundation danese(a)wikimedia.org [1] http://en.wikipedia.org/wiki/Intentional_community

3 2

[HTML5] Improving Commons upload interface
by Christophe Henner 04 Sep '10

04 Sep '10

Hi Everyone, I searched the archives of both list and couldn't find any thread about it. I could miss it, so sorry it it already has been discussed. We all know that uploading a file on commons, and on Wikimedia, is kind of tricky nowdays. However, this could be changed thanks to HTML5. HTML5 includes the drag and drop thingy that makes the uploads easier and that can automatically fetch the EXIF datas. If we want we could even allow the multiple drag and drop. Anyway this could be a solution to look at, you can get more information there : http://hacks.mozilla.org/2009/12/firefox-36-fileapi-demo-reading-exif-data-… and there : http://hacks.mozilla.org/2009/12/file-drag-and-drop-in-firefox-3-6/ All the best, -- Christophe

16 34

← Newer
1
2
3
4
5
6
7
8
9
10
Older →

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l September 2010