Over the last couple of years, MediaWiki development has moved from being almost entirely volunteer-based to having a large contingent of paid developers. A lot of people have noted that this has led to a lot of work being done without much community involvement. Just for a basic statistic, in July, I estimate that about 90% of non-localization commits to extensions/UsabilityInitiative/ were by paid employees. (I use "employee" loosely in this post, to include all paid staff, such as contractors.) By contrast, about 25% (ballpark figure) of non-localization commits to phase3/ were by paid employees, and the number of volunteer commits to phase3/ was much higher than the total number of commits to UsabilityInitiative, so this isn't just a matter of community members not doing as much work overall.
I've commented on this a few times before, but never at length. I think there's widespread confusion about what the problem even is, never mind how to solve it, so I'm writing this to set out at least my own views on the topic. Since my shorter remarks in other places tended to be misunderstood, I'll start at the beginning and go into considerable detail, which means this post will probably end up pretty long. I should say in advance that I'm discussing institutional problems here, not anything specific to individuals or projects, and no one should feel slighted if I pick them as an example. If you aren't really interested, start skimming. ;)
Let me begin with definitions. I will draw a basic distinction between community development and centralized development. I'll start with two motivating examples.
Firefox is developed by a community. Everything involved in the project and its development is open. Most of the work is done by employees of Mozilla, and all important decisions are made by employees of Mozilla, but anyone on the Internet can view what's happening and get involved. Bugs you open might get ignored forever, and you might have to poke people a bunch to get patches reviewed, and you might have to tolerate a considerable amount of bluntness and follow other people's marching orders if you want to contribute anything. But in principle, any random person in the world can make largely the same contributions as a Mozilla employee.
Internet Explorer is developed by a centralized team. They have blogs where they sometimes share detailed info about their development process and reasoning. They very carefully read all user feedback left in the comments. They have a bug tracker where anyone can file bugs, and they guarantee that they'll look at and attempt to reproduce every single bug filed in a timely fashion. But although they pay close attention to feedback, giving feedback is the only way you can really participate without getting hired by Microsoft. You can't write any code, or have a voice in discussions at all comparable to an IE team member.
These examples illustrate some important things:
* Community development does not mean democracy. Even in a totally community-oriented project, all decisions might ultimately be made by a small group of individuals. (For instance, in the case of the Linux kernel, one person.) * Community development does not mean community members do most of the work. From what I've heard, employees of Mozilla write most of Firefox's code, but it's still completely community-oriented development. * Listening to feedback is not the same as actually involving the community. Even a totally closed project can be extremely attentive to feedback. In fact, it's common for community projects to be *less* receptive to feedback, taking a "we'll listen to you when you write the code" attitude.
Keeping these in mind, I'll characterize a perfectly community-based development process like this: your say in the project is proportional to your contributions, and nothing prevents you from contributing as much as your time and ability allow. If you happen to be paid, it doesn't give you any additional say -- you just happen to be able to spend more time contributing. The decision-making process is open and transparent, and arguments are weighed on the basis of their merits and the speaker's history of contributions. This is of course not fully attainable in practice, but one can see how close or far a project is from the ideal.
Centralized and community development processes both have advantages and disadvantages. Some of the advantages of centralized development (as relevant to open-source projects) are:
* Paid employees don't have to spend time reviewing code from a lot of people who will only ever contribute a few patches, so they don't duplicate effort teaching everyone their project's coding conventions, or even educating them on basic things like XSS. * Because discussion can be private and everyone is more likely to be in similar time zones, it's possible to rely heavily on face-to-face or voice communication, which a lot of people are more comfortable with and which is a lot more efficient. * Since there are many fewer developers, they can socialize and get to know each other, reducing conflict and argument. * Full-time developers don't have to try coordinating with volunteers who may only be available at odd times or who may disappear randomly for weeks.
In short, centralized development allows employees' time to be spent more on actual coding, and less on communication. It's (at least superficially) more efficient. On the other hand, community development has advantages as well:
* You get work done for free. If it's easier to volunteers to make a meaningful difference, you'll get many more volunteers. Once they're up to speed, you don't have to watch over them much more than you would an employee, but you get their work for free. * You can hire community developers. You already know how good they are and they don't need to be brought up to speed with your codebase, saving you a lot of money and trouble compared to advertising for applicants. * Your software becomes more versatile, because volunteers will work on aspects that interest them even if they aren't in the interest of the controlling organization. This gets you more users and more developers.
Although there are superficial efficiency advantages to centralizing development, experience indicates that community-based development can be much more cost-effective in practice. Projects like Mozilla and Apache (and for that matter Wikimedia until recently) make software that's very competitive with centrally-developed competitors at a fraction of the cost.
On top of that, of course, the idea of centralized development is contrary to Wikimedia's ideals. Just as the Board is trying to pursue individual donations over corporate sponsorship, it fits with Wikimedia's goals and structure to have as community-oriented a structure as possible. Projects like Mozilla make it clear that this is attainable and productive.
Returning to the concept of community development, let's look at two key things: actual coding, and decision-making. In community-based development, anyone who's willing to write good code can get it submitted and included into the product. Someone with a greater history of contributions will be able to get their code included more easily, but only because the development community is willing to trust them more. They get by with less review, and the review is more readily given because of a greater expectation that it will be productive. Similarly, when it comes to decision-making, anyone has an equal opportunity to try convincing the decision-makers (who might be only one or a few people) of their point of view. In the end, the decision is made by appointed decision-makers, but with great deference toward the opinions of other established contributors.
From my perspective as a volunteer developer since 2006
(notwithstanding a few hours of contracting just now), Wikimedia has been failing badly on both of these issues for months, at least. There's a giant code review backlog, so very little code of the last several months gets synced -- except code by employees. Some employees apparently have shell access for the sole purpose of syncing their own code without going through the normal review process. No volunteer has been given such access, to my knowledge -- indeed, AFAIK it's been years since any non-employee has been given shell access at all. This is a bright line that deprives volunteers of any semblance of parity with staff.
Communication is a serious problem as well. I can't pin this one down so well, because I simply have no idea how employees are communicating, but I can observe that there's a ton of code being written with no discussion on #mediawiki or wikitech-l or any other MediaWiki development forum I know of. There are a lot of paid developers who I've never seen in either #mediawiki or wikitech-l. I infer that they must be communicating somehow, unless they all have a policy of committing code without speaking to anyone about it.
A lot of employees are in the same office, so I guess there's face-to-face communication going on. There's a secret staff IRC channel, and a staff-only mailing list or list alias or something (which I know about because a staff member complained about it in the secret staff IRC channel), and I think I've heard rumor of teleconferences. There are have also been various nominally public fora that only particular groups of employees use much in practice, like the Usability wiki and IRC channel (the latter now kind of discontinued but not really). I don't know, but it doesn't matter in the end. What it amounts to is that volunteers are often completely cut out of planning and design.
That's what leads to things like http://www.mediawiki.org/wiki/Special:Code/MediaWiki/67299. Some people said that maybe that could have been phrased better, or something. But the revert wasn't the problem, it was a symptom of the problem. The problem was that the design was decided on somewhere that volunteers couldn't or wouldn't participate. Of course you revert something that contradicts an agreed-upon design -- the problem is that the agreed-upon design was only agreed upon by a small group of employees. How are volunteers supposed to contribute in that environment, if they don't know what tune they're supposed to be dancing to?
The interlanguage link issue perfectly highlights the problem of mentality we have right now. (I'm not picking on the Usability Initiative particularly here, by the way -- it just provides the most ready examples because it's the largest.) You just need to look at this e-mail: http://lists.wikimedia.org/pipermail/foundation-l/2010-June/058936.html It begins "The Usability team discussed this issue at length this afternoon." The Usability Team is a separate body from the community, which holds its discussions separately. "We listened closely to the feedback and have come up with solution which we hope will work for everyone." Listening to feedback, not discussing the merits of the issue with peers.
What should have happened in that case is that each individual Usability Team member who saw the complaint should have posted their own individual, unrehearsed thoughts as an individual. What actually happened was a quintessentially centralized response: secret internal discussion followed by an official position statement. That is not the way that you treat peers. It's how you treat customers or clients.
I've seen this mentality again and again over the last year or two. One time I was discussing a design issue with a Wikimedia employee in #mediawiki, and after a brief discussion, he said (paraphrased) "Sorry, I need to get back to work." Apparently it's only "work" when you're talking to other employees. http://www.mediawiki.org/wiki/Development_process_improvement draws a clear line at every step between employees and developers. This is not the way to attract or keep a healthy volunteer development community.
The solution is not to increase communication between staff and volunteers. It's to make the distinction as irrelevant as possible to actual development. They're all developers, and some happen to get paid. Specific changes I would propose include:
* Consider what to do about code review. This is pretty much the hardest problem on this list, which is why I don't propose a specific solution here, but there has to be a better solution than "assume a bunch of employees are trusted enough to sync their own code, force everyone else to wait months for central review". * Stop concentrating tech employees in San Francisco. Either have most of them work from home, or perhaps establish other small offices so that they're split up. The point is, make them rely on telecommunication, because if you put people in the same office they'll talk a lot face-to-face, and volunteers simply cannot participate. The purpose of putting people together in an office is so that they work together as a team, and this is exactly what you do *not* want, because volunteers cannot be part of that team. This is the second-hardest problem, or maybe the hardest, and I can't give a full solution for it either. I'd suggest checking with Mozilla about how they do it, because I know they do have offices, but they're a perfect example of community-oriented development. * Explicitly encourage all paid developers to do everything in public and to treat volunteer developers as they would paid ones. I'm not saying this should be enforced in any particular manner, but it should be clearly stated so that everyone knows how things are intended to be. * Shut down the secret staff IRC channel. Development discussion can take place in #mediawiki, ops in #wikimedia-tech, other stuff in #wikimedia or whatever. If users interfere with ops' discussions sometimes in #wikimedia-tech during outages or such, set all sysadmins +v and set the channel +m as necessary. That's worked in the past. * Shut down #wikimedia-dev (formerly #wikipedia_usability, kind of). The explicit purpose of the channel is to allow development discussion with less noise, but "noise" here means community involvement. In community development, you do get a lot more discussion, but that's not something you should try avoiding. In general, use existing discussion fora wherever possible, and if you do fragment them, make sure you don't have too much of a staff-volunteer split in which fora people use. * Don't conduct teleconferences about development, ever. Even if volunteers are invited (are they?), time zones and non-MediaWiki obligations make all synchronous communication much harder for volunteers to participate in. Rely primarily on mailing lists, and secondarily on publicly-logged IRC channels (where at least it's easy to read backscroll). * Stop using private e-mail for development, at least to any significant extent. If there are any internal development mailing lists or aliases or whatever used for development, retire them.
I don't know how seriously these suggestions will be taken in practice by the powers that be, but I hope I've made a detailed and cogent enough case to make at least some impact.
On Fri, Sep 3, 2010 at 9:05 AM, Aryeh Gregor Simetrical+wikilist@gmail.com wrote:
... <I scrolled; but agree with the bits I read>... I don't know how seriously these suggestions will be taken in practice by the powers that be, but I hope I've made a detailed and cogent enough case to make at least some impact.
There is one very, very simple solution.
All changes should either be: a) a patch on bugzilla, reviewed by anyone, and approved _on_bugzilla_ by a tech lead b) the landing of a branch, approved by a tech lead in some public forum.
From that, all else flows.
-- John Vandenberg
Aryeh Gregor wrote:
I don't know how seriously these suggestions will be taken in practice by the powers that be, but I hope I've made a detailed and cogent enough case to make at least some impact.
In large part, the problems and solutions are obvious (you pointed out most of them, and this is hardly the first time this has come up). The issue is that those in power (those who sign the paychecks and employment contracts) are deliberately choosing to ignore these problems and their solutions.
This isn't an "assumption of bad faith" and I won't hear anything of the sort. It's the reality. The problems are obvious; the solutions are obvious. What isn't obvious is why certain people in executive positions have chosen to ignore the problems. It would take a matter of minutes to shutdown the private IRC channels and private mailing lists. It would take one order from one of the members of the executive team for substantive code review and deployment to take place. But the current reality is that if it isn't part of "usability" or fundraising, it doesn't get any type of attention or priority.
MZMcBride
2010/9/3 MZMcBride z@mzmcbride.com:
In large part, the problems and solutions are obvious (you pointed out most of them, and this is hardly the first time this has come up). The issue is that those in power (those who sign the paychecks and employment contracts) are deliberately choosing to ignore these problems and their solutions.
This isn't an "assumption of bad faith" and I won't hear anything of the sort. It's the reality. The problems are obvious; the solutions are obvious. What isn't obvious is why certain people in executive positions have chosen to ignore the problems.
Your allegations that these problems are deliberately being ignored is a serious one, and you may take my word for it (although I'm fairly sure that you won't) that these people definitely care. I think you're wrong in assuming that all these solutions are totally obvious to everyone: serious thought needs to be given to this, and these people have more issues on their mind that just this single one. You are right that there doesn't seem to have been any concrete action or clear statements from people in key positions (say Erik, or, better, Danese) and I very much want that to change. I just disagree with the assertion that they don't care.
It would take a matter of minutes to shutdown the private IRC channels and private mailing lists.
You're assuming this is one of the obvious solutions, which I contend above.
It would take one order from one of the members of the executive team for substantive code review and deployment to take place.
Oh really? So I guess we have dozens of people capable of and available for reviewing and deploying code? We don't. As you have said yourself and Aryeh has pointed out, review and deployment has been a problem for a long time, and if one order could have solved it that would've happened long ago.
Roan Kattouw (Catrope)
Roan Kattouw wrote:
2010/9/3 MZMcBride z@mzmcbride.com:
In large part, the problems and solutions are obvious (you pointed out most of them, and this is hardly the first time this has come up). The issue is that those in power (those who sign the paychecks and employment contracts) are deliberately choosing to ignore these problems and their solutions.
This isn't an "assumption of bad faith" and I won't hear anything of the sort. It's the reality. The problems are obvious; the solutions are obvious. What isn't obvious is why certain people in executive positions have chosen to ignore the problems.
Your allegations that these problems are deliberately being ignored is a serious one, and you may take my word for it (although I'm fairly sure that you won't) that these people definitely care. I think you're wrong in assuming that all these solutions are totally obvious to everyone: serious thought needs to be given to this, and these people have more issues on their mind that just this single one. You are right that there doesn't seem to have been any concrete action or clear statements from people in key positions (say Erik, or, better, Danese) and I very much want that to change. I just disagree with the assertion that they don't care.
I'm not sure if you intended it as such, but this reads as an appeal to emotion. It's not a matter of feelings or a matter of whether someone is committed to Wikimedia's mission or anything of that sort. That is, it isn't about whether people "care" in the sense in which you're using it. It's a matter of whether those in control are making it a priority. And from where I'm sitting, it seems fairly clear that general code review and deployment isn't being made a priority. At least where I work, if my boss says to do something, it gets done.
And, more to the point, how many managers (or other employees) need to be hired before "serious thought" can be devoted to these issues? They seem fairly fundamental to me for an organization that runs websites.
It would take a matter of minutes to shutdown the private IRC channels and private mailing lists.
You're assuming this is one of the obvious solutions, which I contend above.
It seems fairly obvious to me. Aryeh's points about #wikimedia-dev seem fairly spot-on. Do you disagree? If so, why?
It would take one order from one of the members of the executive team for substantive code review and deployment to take place.
Oh really? So I guess we have dozens of people capable of and available for reviewing and deploying code? We don't. As you have said yourself and Aryeh has pointed out, review and deployment has been a problem for a long time, and if one order could have solved it that would've happened long ago.
Yes, really. When it's fundraising-related or Usability-related, there seems to be no issue with code deployment. The server admin log bears me out on this.[1] So yes, I will contend that there would be man-power for review and deployment of the rest of the codebase if it were made a priority.
MZMcBride
All of us at WMF care and follow discussions like this, especially clearly laid out and well thought-out analysis like Aryeh's original post. We don't always agree. :-) I know Danese is planning to weigh in, so I won't write too much at this time, but will point to this post from a couple of months ago where I laid out my view on some (not all) of these topics.
http://lists.wikimedia.org/pipermail/foundation-l/2010-June/059052.html
In general, I think most of us are in favor of more public communications, including public lists, participation in public IRC channels, wikis, etc. I don't agree with unrealistic suggestions (e.g. face-to-face meetings have very real and very serious productivity advantages that we don't want to lose), but we've generally been trending in this direction. Finally, most of these decisions aren't made by "executive fiat" -- Wikimedia is a very collaborative organization, and virtually all the decisions about how/where to communicate and work have been made by the people doing the work.
I would wager a guess that some less constructive individuals on this list and others play a role in what choices people make. :) So, if you're trying to change the dynamics, please call out and help put an end to unconstructive and unprofessional behavior just as much as you ask for public collaboration.
Erik Moeller wrote:
All of us at WMF care and follow discussions like this, especially clearly laid out and well thought-out analysis like Aryeh's original post. We don't always agree. :-) I know Danese is planning to weigh in, so I won't write too much at this time, but will point to this post from a couple of months ago where I laid out my view on some (not all) of these topics.
In general, I think most of us are in favor of more public communications, including public lists, participation in public IRC channels, wikis, etc. I don't agree with unrealistic suggestions (e.g. face-to-face meetings have very real and very serious productivity advantages that we don't want to lose), but we've generally been trending in this direction. Finally, most of these decisions aren't made by "executive fiat" -- Wikimedia is a very collaborative organization, and virtually all the decisions about how/where to communicate and work have been made by the people doing the work.
I realize that management styles can and will differ, but here's the breakdown for posts by Danese to wikitech-l since she was hired in late January / early February:
Feb 2010: 0 Mar 2010: 5 Apr 2010: 0 May 2010: 0 Jun 2010: 2 Jul 2010: 1 Aug 2010: 2
That's ten posts to the central Wikimedia development mailing list in seven months. Are you suggesting Danese is just a very quiet person?
And while I have these tabs open, your stats for the same time period, Erik:
Feb 2010: 0 Mar 2010: 3 Apr 2010: 1 May 2010: 0 Jun 2010: 0 Jul 2010: 1 Aug 2010: 1
That would be six posts in seven months.
Do you think this is acceptable? Do you think it leaves anyone on the outside (or on the inside) with the perception that the Wikimedia technical executive team is concerned with being engaged with the community? When you contrast Danese's stats or your stats with Brion's or Tim's, what do you think the underlying message is?
MZMcBride
On 9/2/2010 9:04 PM, Roan Kattouw wrote:
Oh really? So I guess we have dozens of people capable of and available for reviewing and deploying code? We don't. As you have said yourself and Aryeh has pointed out, review and deployment has been a problem for a long time, and if one order could have solved it that would've happened long ago.
A long time ago, we didn't have this much of a problem with code review (except extensions/branches) and deployment. A couple years ago, we updated the site to trunk at least once a month from what I recall. We still had big features (CentralAuth, preprocessor rewrite) and still ran a fundraiser. The foundation now has probably 3 times as many staff members as then, but from the community's POV seems to get less done.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 37-01--10 03:59 PM, Alex wrote:
The foundation now has probably 3 times as many staff members as then, but from the community's POV seems to get less done.
Seems true to me, except in the case of the Vector project where the red carpet gets laid out. Obviously a concern - especially since the Foundation has consistently said they aim to hire smart instead of fast. Is the tech team really doing better now than they were then?
- -Mike
On Thu, Sep 2, 2010 at 7:05 PM, Aryeh Gregor Simetrical+wikilist@gmail.com wrote:
I don't know how seriously these suggestions will be taken in practice by the powers that be, but I hope I've made a detailed and cogent enough case to make at least some impact.
I hope so as well. You managed to articulate a feeling I share quite deeply. I agree with you on pretty much every point here.
A /very/ strong +1
-Chad
2010/9/3 Aryeh Gregor Simetrical+wikilist@gmail.com:
- Consider what to do about code review. This is pretty much the
hardest problem on this list, which is why I don't propose a specific solution here, but there has to be a better solution than "assume a bunch of employees are trusted enough to sync their own code, force everyone else to wait months for central review".
The current code review situation is a problem, and I don't have a ready-to-go solution for it either. Just wanted to point out I do completely agree with one of your important points before I start disagreeing with parts of almost all your other points.
- Stop concentrating tech employees in San Francisco. Either have
most of them work from home, or perhaps establish other small offices so that they're split up. The point is, make them rely on telecommunication, because if you put people in the same office they'll talk a lot face-to-face, and volunteers simply cannot participate. The purpose of putting people together in an office is so that they work together as a team, and this is exactly what you do *not* want, because volunteers cannot be part of that team. This is the second-hardest problem, or maybe the hardest, and I can't give a full solution for it either. I'd suggest checking with Mozilla about how they do it, because I know they do have offices, but they're a perfect example of community-oriented development.
I personally don't think it should be necessary to enforce discipline (i.e. doing stuff in public) this way. Sometimes, you just want to bounce your ideas off this one person that happens to be an expert in that specific field. To give another example, I did in-person code review at the office with Ryan Kaldari last week, which was very productive. Both are inherently one-on-one and don't need to happen in public: in the bouncing ideas case, you're gonna take it public later after one person has told you it's not totally insane, in the code review case there's barely any benefit to doing it in public because it's really this one guy reviewing revisions written by this other guy.
I also think that we already have a fair number of tech employees outside of San Francisco, and AFAIK we're definitely open to hiring remote people for tech jobs unless in-person interaction is essential, say for a CTO or an EPM (although we do have a half-remote EPM). I do agree that the current level of community participation and feeling like you're part of the community is lacking, but I don't agree with your solutions.
- Explicitly encourage all paid developers to do everything in public
and to treat volunteer developers as they would paid ones. I'm not saying this should be enforced in any particular manner, but it should be clearly stated so that everyone knows how things are intended to be.
I mostly agree here. As I said above I think there's things that don't need to happen in public (little to no added value while raising the bar: walking over to someone's cubicle is easier than broadcasting your possibly stupid idea to the world), but I agree that we're not doing enough in public at this time.
- Shut down the secret staff IRC channel. Development discussion can
take place in #mediawiki, ops in #wikimedia-tech, other stuff in #wikimedia or whatever. If users interfere with ops' discussions sometimes in #wikimedia-tech during outages or such, set all sysadmins +v and set the channel +m as necessary. That's worked in the past.
You're assuming that development discussions actually take place in these channels, which is not the case. We mostly use them for chit-chat and private or office-related things. Questions like "how should I design X in my code" very rarely show up in these channels, and I don't think anyone would have a problem with being more conscious about this and moving to a public place for such a discussion.
- Shut down #wikimedia-dev (formerly #wikipedia_usability, kind of).
The explicit purpose of the channel is to allow development discussion with less noise, but "noise" here means community involvement. In community development, you do get a lot more discussion, but that's not something you should try avoiding. In general, use existing discussion fora wherever possible, and if you do fragment them, make sure you don't have too much of a staff-volunteer split in which fora people use.
No, "noise" means bots and people trying to support people with questions like "how do I disable anon reads on my wiki" as opposed to developers (paid and unpaid alike) being engaged in a design discussion. Maybe #wikimedia-dev should be renamed to #mediawiki-dev to remove the suggestion of WMF-exclusivity, but I definitely see the value of a channel dedicated to communication between developers with support questions and bots kept out.
- Don't conduct teleconferences about development, ever. Even if
volunteers are invited (are they?), time zones and non-MediaWiki obligations make all synchronous communication much harder for volunteers to participate in. Rely primarily on mailing lists, and secondarily on publicly-logged IRC channels (where at least it's easy to read backscroll).
- Stop using private e-mail for development, at least to any
significant extent. If there are any internal development mailing lists or aliases or whatever used for development, retire them.
These points are mostly fair in that the usability team (which I was on) used to have a lot of discussions on the phone and over private e-mail. However, we (features engineering team, which pretty much all the usability folks got moved into) now have a weekly meeting format that focuses more on what each individual has been doing the past week and will be doing the upcoming week and less (still nonzero in some cases, that's something I'm sure Alolita will be willing to fix) on discussions about features.
I don't know how seriously these suggestions will be taken in practice by the powers that be, but I hope I've made a detailed and cogent enough case to make at least some impact.
Some of your points are good, some I disagree with, and some I think are based on paranoia-fed overestimations as to how secretive we're being. I definitely think we need to move more discussions to public places and I will advocate for that to happen, but I don't think we need to take things quite as far as you propose. So let's start by getting the powers that be behind that idea and see if we can improve the situation that way first.
Roan Kattouw (Catrope)
On 2 September 2010 17:40, Roan Kattouw roan.kattouw@gmail.com wrote:
2010/9/3 Aryeh Gregor Simetrical+wikilist@gmail.com:
- Shut down the secret staff IRC channel. Development discussion can
take place in #mediawiki, ops in #wikimedia-tech, other stuff in #wikimedia or whatever. If users interfere with ops' discussions sometimes in #wikimedia-tech during outages or such, set all sysadmins +v and set the channel +m as necessary. That's worked in the past.
You're assuming that development discussions actually take place in these channels, which is not the case. We mostly use them for chit-chat and private or office-related things. Questions like "how should I design X in my code" very rarely show up in these channels, and I don't think anyone would have a problem with being more conscious about this and moving to a public place for such a discussion.
I don't think he is making that assumption, lots of chit-chat happens on any channel — it makes us feel left out if you have a special one (humans are really bad at the jealousy thing). What "private or office-related" things are there that you can't share with developers?
Conrad
On 03.09.2010, 4:40 Roan wrote:
- Shut down #wikimedia-dev (formerly #wikipedia_usability, kind of).
The explicit purpose of the channel is to allow development discussion with less noise, but "noise" here means community involvement. In community development, you do get a lot more discussion, but that's not something you should try avoiding. In general, use existing discussion fora wherever possible, and if you do fragment them, make sure you don't have too much of a staff-volunteer split in which fora people use.
No, "noise" means bots and people trying to support people with questions like "how do I disable anon reads on my wiki" as opposed to developers (paid and unpaid alike) being engaged in a design discussion. Maybe #wikimedia-dev should be renamed to #mediawiki-dev to remove the suggestion of WMF-exclusivity, but I definitely see the value of a channel dedicated to communication between developers with support questions and bots kept out.
It should ultimately be decided whether MediaWiki is a software developed by an open community for everyone to use, or just a software that runs Wikipedia and is sporadically released to public only for lulz and Wikia? If it's the latter, such "noise" is indeed unwanted. If the former, this chit-chat is vital to being a part of the community.
On Fri, Sep 3, 2010 at 11:54 AM, Max Semenik maxsem.wiki@gmail.com wrote:
On 03.09.2010, 4:40 Roan wrote:
- Shut down #wikimedia-dev (formerly #wikipedia_usability, kind of).
The explicit purpose of the channel is to allow development discussion with less noise, but "noise" here means community involvement. In community development, you do get a lot more discussion, but that's not something you should try avoiding. In general, use existing discussion fora wherever possible, and if you do fragment them, make sure you don't have too much of a staff-volunteer split in which fora people use.
No, "noise" means bots and people trying to support people with questions like "how do I disable anon reads on my wiki" as opposed to developers (paid and unpaid alike) being engaged in a design discussion. Maybe #wikimedia-dev should be renamed to #mediawiki-dev to remove the suggestion of WMF-exclusivity, but I definitely see the value of a channel dedicated to communication between developers with support questions and bots kept out.
Have to say this is the first I've heard of this channel existing. Yay for communication.
On Fri, Sep 3, 2010 at 2:06 PM, OQ overlordq@gmail.com wrote:
Have to say this is the first I've heard of this channel existing. Yay for communication.
It was only created very recently.
On Fri, Sep 3, 2010 at 2:14 PM, Chad innocentkiller@gmail.com wrote:
As I've said elsewhere to people, this isn't an excuse for fracturing the discussion. Using a single channel for development *and* support has worked for *years* until the Usability Initiative decided it needed its own channel.
The channel is not actually that busy if you don't count the bots. And for those of you who *really* don't like them, you can /ignore them. I don't consider people asking for help "noise" either, it's part of being engaged with the community.
I don't see the need for a separate development channel at all.
Me neither. We don't get that many support questions.
On Fri, Sep 3, 2010 at 4:14 PM, Neil Kandalgaonkar neilk@wikimedia.org wrote:
Aryeh: first off I want to thank you again for the constructive criticism.
And thank you for the thoughtful and constructive response.
I think you are not really appreciating that the WMF employees are also human beings. We share an office. We go out to lunch together every day.
This is, as I said, one of the problems -- at least from the point of view of greater volunteer involvement. (That you share an office and go to lunch together, not that you're human beings. :) ) It's not necessarily prohibitive, in that you could certainly have totally healthy community development with lots of paid developers in one place, but it definitely lends itself to locking out volunteers.
I don't see anyone deliberately hiding things.
It's more the case that we don't have established procedures about how to be open, in a regular, repeatable fashion. We try really hard but the efforts are always competing with just trying to get things done.
I don't know what you can call a private IRC chat where staff gets to hang out and non-staff does not, except deliberately hiding things. I can't speak for anything else because, well, it's hidden. I know it's hidden somewhere, because I don't see it, but I don't know where, or whether it's deliberate or not.
And in part that openness has practical limits, for exactly the same reasons that a network card has limited bandwidth.
There's no practical limit here. An approach where everyone communicates almost entirely by public e-mail is feasible. Any open-source project without a single well-funded controlling organization works that way by necessity -- the Linux kernel, Debian, etc. Some with such a controlling organization also work that way.
When Wikipedia started, it wasn't a perfectly distributed effort either; they always had developers and people like Jimmy Wales in the same room.
This certainly wasn't true in 2006, when I got commit access. There were two paid developers at the time, both of whom lived more than a thousand miles from Jimmy Wales. At that time there was no office at all, IIRC, but if there was, it was in Tampa -- Brion lived in San Francisco and Tim in Australia.
It is my impression that the reason they are able to be so open and communicate well about roadmaps and so on is because they do have enough resources (centralized) to do such coordination and make sure to publish documents and do press releases and whatnot. If everyone was dispersed I don't think they'd be very successful at this task.
Everyone *is* dispersed. For instance, the layout engine is owned by Robert O'Callahan (New Zealand), and its peers are Bernd Mielke (?), Simon Montagu (Israel), L. David Baron (Los Angeles area), Boris Zbarsky (Boston area). (Locations based on Googling, might not all be accurate.) Linked from the mozilla.com website:
""" Most positions are based at our headquarters in Mountain View, California, but we also have offices in Tokyo, Paris, Toronto, Beijing and Auckland (with more to come!). Not near one of our offices? Mozilla thrives as an organization because of our diverse and distributed workforce, so remote employment is an option. """ http://hire.jobvite.com/CompanyJobs/Jobs.aspx?c=qpX9Vfwa
As far as I can tell, very little happens at Mozilla face-to-face, compared to what's done by newsgroup/Bugzilla/IRC. I've subscribed to lots of Mozilla bugs and watched patches get submitted and reviewed, and I've never noticed much of anything that seemed to be hidden. Maybe because the patch author and reviewer usually live in separate parts of the world. Even if Mozilla weren't a perfect example, though, the Linux kernel is a case where virtually nothing important happens except by mailing list, which proves it's feasible in principle to run a large software project that way.
I'm okay with discussing whether there might be efficiency or morale problems with not having most paid developers in a central office, but it's just not correct to suggest that it's impractical to develop software that way. It's entirely practical. You can efficiently produce high-quality software with practically all communication online and public.
On Fri, Sep 3, 2010 at 9:05 PM, Neil Kandalgaonkar neilk@wikimedia.org wrote:
To his credit Aryeh is aware of this but he believes that the productivity hit is worth it if it ensures the organization is 100% transparent. I just don't agree we need to go to such lengths.
I made seven suggestions. Only one was about actually dissolving the office, and I acknowledged that it might be extreme. What about the others? Why does the private IRC chat need to exist, for example? Why can't we have clear official statements that everything should be as public as possible and that volunteer developers should be treated as peers? Why can't teleconferences be replaced by public-logged IRC chats? Are these also too extreme?
I think you are not really appreciating that the WMF employees are also human beings. We share an office. We go out to lunch together every day.
This is, as I said, one of the problems -- at least from the point of view of greater volunteer involvement. (That you share an office and go to lunch together, not that you're human beings. :) ) It's not necessarily prohibitive, in that you could certainly have totally healthy community development with lots of paid developers in one place, but it definitely lends itself to locking out volunteers.
I don't think you understand the importance of occasionally doing things with people face to face. A lot of trolling, misplaced paranoia, and WMF hate would disappear if *more* people met face to face more often. Working remotely, and never meeting people in real life simply isn't easy. Some people have the mentality to do so, but most people don't. (Most) People are social beings, and text communications strip most elements of being social away. What you are suggesting would make hiring more difficult, and would make working for the foundation more difficult.
Note that I am employed as a remote employee.
I don't see anyone deliberately hiding things.
It's more the case that we don't have established procedures about how to be open, in a regular, repeatable fashion. We try really hard but the efforts are always competing with just trying to get things done.
I don't know what you can call a private IRC chat where staff gets to hang out and non-staff does not, except deliberately hiding things. I can't speak for anything else because, well, it's hidden. I know it's hidden somewhere, because I don't see it, but I don't know where, or whether it's deliberate or not.
The chat in those channels isn't anything crucial or even related to development. If the channels go away, the talk that occurs there will simply move to private chats on IM. This being on IRC just makes it slightly easier to broadcast things to staff members. Do you really care to see chatter in the mediawiki channel like: "Free brownies in the break room.", "Water machine is getting fixed.", "Phone system is getting upgraded."? Should non-tech related employees be forced to deal with all the bot traffic, dev talk, and support talk of #mediawiki? Maybe it doesn't need to be private, but dealing with trolls gets old pretty fast.
As far as I can tell, very little happens at Mozilla face-to-face, compared to what's done by newsgroup/Bugzilla/IRC. I've subscribed to lots of Mozilla bugs and watched patches get submitted and reviewed, and I've never noticed much of anything that seemed to be hidden. Maybe because the patch author and reviewer usually live in separate parts of the world. Even if Mozilla weren't a perfect example, though, the Linux kernel is a case where virtually nothing important happens except by mailing list, which proves it's feasible in principle to run a large software project that way.
A lot of Linux kernel development is done in private email, private channels, face to face, etc. A large amount of kernel code comes from private entities like Red Hat and Novell. Do you participate in these projects nearly as much as MediaWiki? I think your viewpoint is likely skewed here.
I'm okay with discussing whether there might be efficiency or morale problems with not having most paid developers in a central office, but it's just not correct to suggest that it's impractical to develop software that way. It's entirely practical. You can efficiently produce high-quality software with practically all communication online and public.
You can, yes, and we should strive for as much public communication as possible. I'm with you here. I personally don't think the problem is as wide-spread as you though. I've been working on contract with the foundation for the past year, and I don't think I've had any more insight on projects than I did previously (and I've been active since 2004). Only recently have I been added to the private lists and channels, and there is very little there that isn't also on public channels. The posts on the private lists are generally things that can't be posted in public because the material is sensitive in nature.
You complained about this problem a few months ago when I posted to wikitech-l about Selenium. Funny thing is, I was asked to build a Selenium grid infrastructure to help the contracted QA enginneers, and that post was the beginning of the work. You assumed something was occurring in secret. The framework came from a volunteer, and two of the five people that phone into the meetings are volunteers. The first post on Selenium was asking for volunteer participation, and early on we decided all correspondence should be via wikitech-l. All planning is being done via mediawiki.org.
To his credit Aryeh is aware of this but he believes that the productivity hit is worth it if it ensures the organization is 100% transparent. I just don't agree we need to go to such lengths.
I made seven suggestions. Only one was about actually dissolving the office, and I acknowledged that it might be extreme. What about the others? Why does the private IRC chat need to exist, for example? Why can't we have clear official statements that everything should be as public as possible and that volunteer developers should be treated as peers? Why can't teleconferences be replaced by public-logged IRC chats? Are these also too extreme?
I think it is too extreme, yes. Some people are more comfortable talking problems out.
There is a problem with teleconferences, though. The problem is that we don't put enough effort in to inviting volunteers, and don't put enough effort into posting the meeting minutes after the teleconference is done. This needs to be fixed. My proposed solution is:
1. Try to pick a convenient time for the teleconference, based on those who are interested in the content 2. Post the meeting time, number, and agenda well in advance, and encourage volunteers to attend 3. Invite everyone to add/modify the agenda so that their topic can be discussed; agenda items should have a time allocated to them 4. Stick strictly to the meeting agenda during the teleconference, especially in regards to time 5. Mute those who troll/are unwilling to stick to the agenda (they will still be invited to listen) 6. Post minutes as quickly as possible after the meeting is over (hopefully within a couple of hours)
There needs to be some middle ground in regards to teleconferences. What you are proposing is something that you are comfortable with. It is a style you prefer to work. It excludes those who are more comfortable or more productive working in another way.
Respectfully,
Ryan Lane
2010/9/6 Ryan Lane rlane32@gmail.com:
The chat in those channels isn't anything crucial or even related to development. If the channels go away, the talk that occurs there will simply move to private chats on IM. This being on IRC just makes it slightly easier to broadcast things to staff members. Do you really care to see chatter in the mediawiki channel like: "Free brownies in the break room.", "Water machine is getting fixed.", "Phone system is getting upgraded."? Should non-tech related employees be forced to deal with all the bot traffic, dev talk, and support talk of #mediawiki? Maybe it doesn't need to be private, but dealing with trolls gets old pretty fast.
There is also productive, work-related chat going on there, mostly someone working from home (a lot of of our staff have a habit of working from home one day a week) trying to grab someone else. Conversations where all participants work in tech frequently happen in other channels instead.
- Mute those who troll/are unwilling to stick to the agenda (they
will still be invited to listen)
I'm not completely sure that this is easy to do with our phone system, but maybe Skype or WebEx allow it.
- Post minutes as quickly as possible after the meeting is over
(hopefully within a couple of hours)
This should be trivial, as such meeting notes are usually (almost always for meetings I'm in) written collaboratively and on the fly in Etherpad, and publicly readable and editable to everyone who knows the URL (probably best to move stuff to the wikis anyway, though). We often posted the Etherpad link to our usability progress meeting notes in #wikipedia_usability , although to be fair that wasn't done with transparency or non-staff involvement in mind.
Roan Kattouw (Catrope)
I made seven suggestions. Only one was about actually dissolving the office, and I acknowledged that it might be extreme. What about the others? Why does the private IRC chat need to exist, for example? Why can't we have clear official statements that everything should be as public as possible and that volunteer developers should be treated as peers? Why can't teleconferences be replaced by public-logged IRC chats? Are these also too extreme?
Aryeh, I think many volunteer and more casual developers share your concern. I in principle agree with your proposals, although of course no-one can be forced to abandon private communication, and private means of communication are always going to exist. I have raised similar concerns about volunteers not knowing what is going on by not being on secret channels of communication, in person, to a couple of members of staff during last two wikimanias and developer meetings, and I had the feeling they agreed with me.
The community needs to be nurtured, and I think all new employees of the WMF need to be aware of it, and at first interview informed that they will *need* to interact with the community and with volunteer developers. I think many programmers who have worked in programming companies are too used to just talking and listening to their team leaders and no-one else. It should be made clear that this is not how things (should) work in WMF, and this should be an official position from however is hiring. Or maybe it is utopia and we do need to have a more stereotypical corporate setup, but I really hope not because wikipedia is fueled by enthusiasm.
Finally, speaking as a volunteer who is not on any secret IRC channels, mailing lists or payrolls I want to share my experience in WMF software development. Back in 2006 I wanted to make search better, and if then it wasn't for Tim Starling to give me shell access and a couple of test servers to play with, I think we would not have the new search, or at least not developed by me. An act of kindness, but also a sign that a core developer cares about what a relatively unknown volunteer is trying to do and achieve.
As for code review, I know the foundation knows how important this task is, and that it is no 9-5 job, but one that requires an extremely dedicated person with a great knowledge of the mediawiki codebase and ability to comment on virtually every programming issue. The foundation better pay this person well and not just hope for someone to fill in this place in their spare time.
Cheers, Robert
On 9/7/10 4:15 PM, Robert Stojnic wrote:
The community needs to be nurtured, and I think all new employees of the WMF need to be aware of it, and at first interview informed that they will *need* to interact with the community and with volunteer developers.
Just FYI, this was the most-stressed topic during my interviews at the WMF. So I don't think it's a matter of the WMF not being aware of the topic.
I think many programmers who have worked in programming companies are too used to just talking and listening to their team leaders and no-one else.
This is true, but we are trying not to hire this sort of person.
Off the top of my head I would guess at least 75% of the developers here had significant open source experience before being hired, many with Wikimedia projects.
I am both a long-time community member and a new WMF paid developer (in the SF office) so I think I'm in a unique position to clear up some misconceptions.
First of all, all this talk of secret listservs and IRC channels is malarkey. Yes, there are private listservs and IRC channels. All of them are private for very specific and well-established reasons. Most of them are only used in very specific circumstances (for example if there was a security breach that needed to be discussed privately) and tend to be very low traffic. They are not the places where important decisions are made.
Secondly, the idea that developers here in the office don't interact with the community is absurd. The developers here interact with the community constantly. We discuss community opinion and ideas, we talk with community members all day long on IRC, listservs, and on-wiki. I'm amazed that the developers here ever get anything done considering how much time they spend documenting what they are working on and interacting with the community about it. The problem is they can't interact with everyone everywhere: Code Review, IRC, listservs, the Tech Blog, meta, Signpost, etc. So no matter what, someone is going to feel like they are out of the loop.
The WMF is extremely interested in new developers interacting with the community, indeed they try to hire developers from within the community when possible. The notion that the foundation is hiring corporate drones who are only going to listen to their task masters is completely unfounded. Yes, there have been situations where the foundation has been given grant money for very specifically defined projects and those projects have been implemented without adequate community involvement. Everyone (including the foundation) knows that that's not how we want to do development in the future. As has been discussed throughout the past year, the foundation wants to move away from accepting any money with strings attached and away from relying on grants in general. Hopefully, if this year's fundraiser goes well, we won't have to worry about these issues in the future.
Ryan Kaldari
On 9/7/10 4:15 PM, Robert Stojnic wrote:
I made seven suggestions. Only one was about actually dissolving the office, and I acknowledged that it might be extreme. What about the others? Why does the private IRC chat need to exist, for example? Why can't we have clear official statements that everything should be as public as possible and that volunteer developers should be treated as peers? Why can't teleconferences be replaced by public-logged IRC chats? Are these also too extreme?
Aryeh, I think many volunteer and more casual developers share your concern. I in principle agree with your proposals, although of course no-one can be forced to abandon private communication, and private means of communication are always going to exist. I have raised similar concerns about volunteers not knowing what is going on by not being on secret channels of communication, in person, to a couple of members of staff during last two wikimanias and developer meetings, and I had the feeling they agreed with me.
The community needs to be nurtured, and I think all new employees of the WMF need to be aware of it, and at first interview informed that they will *need* to interact with the community and with volunteer developers. I think many programmers who have worked in programming companies are too used to just talking and listening to their team leaders and no-one else. It should be made clear that this is not how things (should) work in WMF, and this should be an official position from however is hiring. Or maybe it is utopia and we do need to have a more stereotypical corporate setup, but I really hope not because wikipedia is fueled by enthusiasm.
Finally, speaking as a volunteer who is not on any secret IRC channels, mailing lists or payrolls I want to share my experience in WMF software development. Back in 2006 I wanted to make search better, and if then it wasn't for Tim Starling to give me shell access and a couple of test servers to play with, I think we would not have the new search, or at least not developed by me. An act of kindness, but also a sign that a core developer cares about what a relatively unknown volunteer is trying to do and achieve.
As for code review, I know the foundation knows how important this task is, and that it is no 9-5 job, but one that requires an extremely dedicated person with a great knowledge of the mediawiki codebase and ability to comment on virtually every programming issue. The foundation better pay this person well and not just hope for someone to fill in this place in their spare time.
Cheers, Robert
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On 2010-09-07, Ryan Kaldari wrote:
I am both a long-time community member and a new WMF paid developer (in the SF office) so I think I'm in a unique position to clear up some misconceptions.
First of all, all this talk of secret listservs and IRC channels is malarkey. Yes, there are private listservs and IRC channels. All of them are private for very specific and well-established reasons. Most of them are only used in very specific circumstances (for example if there was a security breach that needed to be discussed privately) and tend to be very low traffic. They are not the places where important decisions are made.
This has been repeated several times in this thread. But from a volunteers perspective it is difficult to imagine where discussion between staff members is going on then. Until after this topic started I have seen very little discussion of staff projects on this mailing list, no discussion on the IRC channel, and very little on the MediaWiki wiki. These are three of the main enshrined development channels, and up until this discussion of staff activity on them was minimal. I find it difficult to believe that this is all the discussion that is going on, or is everything else in face to face meetings - if so where are the minutes and notes for these, because MediaWiki.org seems the obvious place to put them? And furthermore where is all the project documentation that you say has been produced?
Sorry, but from the point of view of a volunteer the only logical reason there is/was no activity in any of these development channels is that there must be various private ones.
Secondly, the idea that developers here in the office don't interact with the community is absurd. The developers here interact with the community constantly. We discuss community opinion and ideas, we talk with community members all day long on IRC, listservs, and on-wiki. I'm amazed that the developers here ever get anything done considering how much time they spend documenting what they are working on and interacting with the community about it. The problem is they can't interact with everyone everywhere: Code Review, IRC, listservs, the Tech Blog, meta, Signpost, etc. So no matter what, someone is going to feel like they are out of the loop.
This may well be true for the community in general, but this discussion is specifically about the volunteer developer community, which is clearly being left out of the loop in a large respect - otherwise this discussion would not exist.
They are arguably the most important members of the community from a technology perspective as volunteer developers have the time and initiative to actively improve MediaWiki, and development cannot survive on ideas and feedback alone. By readily involving the dev. community the WMF creates an environment where there is the potential (not everything will recieve a community response, but the potential for it then exists) for more ideas are implemented cost-free, and paid development time can be focused more on what it needs to be to maximise gain for all stakeholders.
The WMF is extremely interested in new developers interacting with the community, indeed they try to hire developers from within the community when possible. The notion that the foundation is hiring corporate drones who are only going to listen to their task masters is completely unfounded. Yes, there have been situations where the foundation has been given grant money for very specifically defined projects and those projects have been implemented without adequate community involvement. Everyone (including the foundation) knows that that's not how we want to do development in the future. As has been discussed throughout the past year, the foundation wants to move away from accepting any money with strings attached and away from relying on grants in general. Hopefully, if this year's fundraiser goes well, we won't have to worry about these issues in the future.
There is nothing wrong with recieiving technology grants for specific projects, and with the correct transparency and integration I think they can be a good thing as they are likely to improve areas that were previously ignored and bring new developers, and their insight, on to the paid staff.
Transparency and integration were the main things the usability initiative lacked, but on their whole the contribution to MediaWiki and the WMF projects is undoubtedly very large.
Robert
2010/9/7 Robert Leverington robert@rhl.me.uk:
if so where are the minutes and notes for these, because MediaWiki.org seems the obvious place to put them?
Indeed, there are lots of face-to-face meetings / teleconferences where minutes are currently captured on EtherPad, but I don't think these are routinely visibly shared. I think it would be great, wherever possible, to post these to MediaWiki.org in standardized places so that folks can follow what's happening (and express interest in participating). And while we've stopped using usability@wm, there's still quite a bit of e-mail traffic that could be usefully directed toward wikitech-l or to lists that are, at minimum, publicly archived and have clear processes for joining and leaving.
You and Ryan both right -- there's still too much of an in-group/out-group communication pattern, and too little explicit invitation of and collaboration with volunteers. As I hope recent communications demonstrate, that's slowly shifting, but it will take some time. And yes, it takes threads like this one. But, Ryan is correct that there's no pattern of deliberate secrecy here either, and if you developed an open source transparency/collaboration scorecard for WMF, you'd probably get a mixed result today. We're doing well in some areas like general corporate transparency [1], or making sure that all code goes into a public VCS, or granting commit access liberally, or being routinely and openly communicative in countless public spaces and at all levels. But there's no reason we shouldn't be best in every category. ;-) And that ultimately means seeing _everything_ as a shared responsibility.
Everyone who works for Wikimedia, as a volunteer or staff member, is passionate about what we're doing, or they won't be here for long. We're here to build wonderfully useful information resources and the open source technologies to sustain and grow them -- and to be excellent at it. If we maintain awareness of the forcing functions that influence how people communicate, then I think that's eminently achievable. This includes, for example:
- felt deadline pressure in the original usability project - problems with signal/noise ratio on existing lists / channels - in-group bias created by high proximity / peer relationships among WMF staff
This isn't a game of assigning responsibilities or blame. Rather, it's about us collectively breaking out of the in-group/out-group pattern, creating constructive and healthy public spaces of communication and collaboration, remaining flexible about deadlines and targets where possible, reminding ourselves to be inclusive and open in how we work, etc. I'm confident that we can do it. :-)
Erik
[1] It's a fun exercise to research other organizations, non-profits and for-profits. I do so routinely as part of my work, and there are very, very few similarly funded organizations that even come close to the level of disclosure that WMF is currently at.
On 9/7/10 11:23 PM, Robert Leverington wrote:
I find it difficult to believe that this is all the discussion that is going on, or is everything else in face to face meetings - if so where are the minutes and notes for these, because MediaWiki.org seems the obvious place to put them? And furthermore where is all the project documentation that you say has been produced?
For my work specifically (which isn't even of much interest to the dev community), you can find it documented and discussed at: http://meta.wikimedia.org/wiki/CentralNotice_upgrades http://www.mediawiki.org/wiki/Extension:CentralNotice/Phase_2 http://www.mediawiki.org/wiki/Extension:CentralNotice/Phase_3 http://techblog.wikimedia.org/2010/09/wmf-engineering/#more-1006
And most of those features are discussed in individual Bugzilla requests.
This may well be true for the community in general, but this discussion is specifically about the volunteer developer community, which is clearly being left out of the loop in a large respect - otherwise this discussion would not exist.
I agree that the volunteer dev community is not as in-the-loop as paid staff, but it's not because the Foundation is trying to keep them out of the loop. Communicating with the dev community, the broader community, fellow staff, and keeping up with an aggressive development schedule is not an easy task! It doesn't take a conspiracy to not satisfactorily fulfill all of those requirements. Obviously, we can stand some improvement, which is why the Foundation is planning on specifically hiring someone to act as a bugmeister and development coordinator in the very near future. What we need are helpful (and realistic) suggestions for how to improve communication, not misguided badgering about "secret channels".
Ryan Kaldari
On 9/8/2010 1:45 PM, Ryan Kaldari wrote:
On 9/7/10 11:23 PM, Robert Leverington wrote:
This may well be true for the community in general, but this discussion is specifically about the volunteer developer community, which is clearly being left out of the loop in a large respect - otherwise this discussion would not exist.
I agree that the volunteer dev community is not as in-the-loop as paid staff, but it's not because the Foundation is trying to keep them out of the loop. Communicating with the dev community, the broader community, fellow staff, and keeping up with an aggressive development schedule is not an easy task!
In 90% of of cases, there should be little difference between communicating with staff and communicating with the developer community. Obviously there will be some non-development related staff communication, but there really shouldn't be a difference between communicating with staff and the community. An intentional or incidental separation between staff and community is really the underlying problem.
It doesn't take a conspiracy to not satisfactorily fulfill all of those requirements. Obviously, we can stand some improvement, which is why the Foundation is planning on specifically hiring someone to act as a bugmeister and development coordinator in the very near future. What we need are helpful (and realistic) suggestions for how to improve communication, not misguided badgering about "secret channels".
Getting rid of excessive amounts of communication channels isn't a useful suggestion? There are 2 main public mailing lists and 3 main public IRC channels. Then there's about a dozen other "minor" public mailing lists. How many additional ones are needed? Why does the non-staff community need to be excluded from them?
A few years ago, there were few staff developers and many worked remotely. There were few of the communication issues we have now. More recently, the WMF hired more staff developers and began concentrating them in the main office and some private channels/lists sprung up to support them. Now we have communication problems. Its possible its just a coincidence, but it certainly doesn't look like one. At the very least, decreasing excessive "bureaucracy" (for lack of a better term) is something that should be considered as a matter of course.
Hi!
Back in 2006 I wanted to make search better, and if then it wasn't for Tim Starling to give me shell access and a couple of test servers to play with, I think we would not have the new search, or at least not developed by me.
I probably have similar story to share :)
Unfortunately, on the infrastructure/operations engineering side, there has been a shift to this whole new "we are a major website that should never go down" concept, which restricts volunteers from getting immediate shell accesses. On one hand, some of us have been lucky to join when there were no such constraints and could go through all the learning process, on another - we wouldn't get such access nowadays.
I quite enjoyed when our site SLA was "more up than down", or you know, when none of us had many interests outside having this damn thing work :)
In commercial organizations besides signing "I won't do crap" contracts, there's also "loss of income" incentive that protects from doing crap. Our infrastructure (and infrastructure team) is way smaller than all the aspirations around of "being a top site". How would we match volunteer contributions in this field with overall demands?
Domas
P.S. There have been way more folks telling "pay me!" than "how can I volunteer" in the infra side, than on mw dev, I guess ;-)
On Thu, Sep 2, 2010 at 8:40 PM, Roan Kattouw roan.kattouw@gmail.com wrote:
I also think that we already have a fair number of tech employees outside of San Francisco, and AFAIK we're definitely open to hiring remote people for tech jobs unless in-person interaction is essential, say for a CTO or an EPM (although we do have a half-remote EPM). I do agree that the current level of community participation and feeling like you're part of the community is lacking, but I don't agree with your solutions.
What solutions do you propose?
You're assuming that development discussions actually take place in these channels, which is not the case. We mostly use them for chit-chat and private or office-related things. Questions like "how should I design X in my code" very rarely show up in these channels, and I don't think anyone would have a problem with being more conscious about this and moving to a public place for such a discussion.
I'm not assuming that -- I've been idling in the secret channel for a while now. (I keep almost saying its name, argh. Channels that aren't access-restricted and rely on secret names are annoying.) Most of it is just chitchat. But that's exactly something that the broader community should be part of, so that staff doesn't form its own group that excludes volunteers. 95% of it could easily be public, and the rest could easily be taken to private e-mail or PM. There is not enough stuff that needs to be private to justify a whole channel.
Some of your points are good, some I disagree with, and some I think are based on paranoia-fed overestimations as to how secretive we're being.
Let me put it this way: I am saying what I perceive as a volunteer. If the goal is to make volunteers feel like they're a full-fledged part of the development community, then the fact that I made this post and the fact that every single volunteer who's commented so far appears to basically agree with me means that ipso facto, my complaints are valid.
You're looking at it from the perspective of someone who sees all this stuff. "Oh, don't worry, it's nothing important." Well, thank you for reassuring me -- but I'm capable of deciding myself whether something is important, if I can see it. Maybe I'll find some of it important. But I'm given no opportunity. Nor is any other volunteer. The problem isn't necessarily the actual content of what we can't see, but the mere fact that so much is clearly hidden. It draws a line that need not exist.
When you hide things from volunteers, you cannot turn around and blame them for inaccurate speculation about what you've hidden. If you don't want speculation (or paranoia, as you put it), there's an easy solution: make it public. Anything short of that is not really going to solve the problem.
On Thu, Sep 2, 2010 at 8:08 PM, MZMcBride z@mzmcbride.com wrote:
In large part, the problems and solutions are obvious (you pointed out most of them, and this is hardly the first time this has come up). The issue is that those in power (those who sign the paychecks and employment contracts) are deliberately choosing to ignore these problems and their solutions.
I will reply to you in this thread only to say that as usual, you are being aggressive and unconstructive, and your input is unwelcome. Personally, I didn't even read any of your posts, or the responses to them, because I knew from long experience that it would be a waste of time. Part of building a successful community is allowing anyone the chance to speak, and part of building a successful community is evicting those who took their chance and blew it. Your contributions consistently clock in only moderately above the level of unconstructiveness that would justify actually banning you, and your endorsement of the point I'm making here will do nothing but harm it. You will not accomplish anything positive by attacking against the people who are responsible for fixing the problem, even if you were actually correct about the cause of the problem (which you are not). My advice to you right now is that the most useful thing you can do is not comment in this thread again.
On Thu, Sep 2, 2010 at 9:20 PM, Erik Moeller erik@wikimedia.org wrote:
I don't agree with unrealistic suggestions (e.g. face-to-face meetings have very real and very serious productivity advantages that we don't want to lose)
But somehow most open-source projects do very well without them, including projects much bigger and better-funded than MediaWiki. I did try to emphasize this in my initial post, but several people apparently missed it. Do you think Wikimedia could do better than to emulate any of the high-profile projects that use almost no face-to-face conversation?
but we've generally been trending in this direction.
My observation as a volunteer is the opposite.
Finally, most of these decisions aren't made by "executive fiat" -- Wikimedia is a very collaborative organization, and virtually all the decisions about how/where to communicate and work have been made by the people doing the work.
Except that volunteers can't do the work on some projects, because we're treated as outsiders and aren't interested in committing lest we be summarily reverted. Platonides' reverted usability commit (even if later reverted back) only confirmed the suspicion that a lot of us held and still hold, that participating in paid projects is a waste of time. I'm not the only one who feels that way, by far. You cannot claim to be a bottom-up meritocracy when a large segment of developers are excluded from the bottom to begin with.
Put it this way: what you're saying here would be akin to allowing a Wikipedia to disabling editing by anonymous users. Sure, it's a bottom-up decision, but it's one that's antithetical to the way that Wikimedia projects are supposed to run. Such requests have been made, and they were always denied (to my knowledge). If you're going to say that Wikimedia wants to allow more centralized development because you think it's more efficient, or something -- fine. That's your decision to make. But please don't say that it's not up to you to decide.
On Fri, Sep 3, 2010 at 1:20 AM, Tim Starling tstarling@wikimedia.org wrote:
Your recommendations seem insensitive and unrealistic. What works for you does not necessarily work for everyone.
It works for many, many open-source projects. Does Wikimedia want to be among them, or does it want to be among the projects that are open-source but have a stagnant community because they don't involve volunteers enough and the code is tightly controlled by a sponsoring organization? There are countless projects of both types -- both strategies work, to a degree. Completely closed-source development works too. Wikimedia is free to pick whichever model best fits its needs and ideals.
On Fri, Sep 3, 2010 at 3:51 AM, Danese Cooper dcooper@wikimedia.org wrote:
I'm not going to deconstruct your suggestions; I actually agree with several of them (although I think you'd find the reality of flagship open source projects somewhat different than the legend).
I can say that despite being a nobody at Mozilla and having gotten only one (rather trivial) patch accepted, I feel like I'm taken more seriously by most of their paid developers than by most of ours. I know more about what Mozilla is planning to do in their next release than what Wikimedia employees are planning. I've had lengthy discussions on occasion with core Mozilla developers, and sometimes I've gotten something changed because of it. I can say none of these things about any Wikimedia project that's dominated by paid employees.
Gradually I decided these were the most serious problems (in order of importance) to tackle with a new design WMF Tech:
- Eliminate single points of failure / bottlenecks
- Reconfigure into teams designed to encourage faster (shorter
duration) and more accurate projects / deployments 3. Develop programs to encourage / grow volunteers into paid staff...recognizing that as a non-profit WMF needs to plan for more frequent turnover in the Tech department to ensure that we can grow expertise across the dev community rather than concentrating it in a few core people. 4. Restore the balance between community-led and foundation-led efforts so WMF feels again like a partnership rather than concentric circles of permission / blame
I can agree with that list of priorities. Wikimedia has a bunch of problems right now, and 3-4 are definitely less important than 1 (not sure about 2). I think there are changes that could be made right now with little administrative effort that would greatly improve 3-4, which I've outlined -- all of my proposals but the first two. I'm somewhat disappointed that no one of importance took any of them seriously. But if you say you'll get to the issue later, well, I can hope.
Let me ask one thing, though. When you do get to the questions of volunteer involvement, start out by ignoring everything the paid people think or say. Compile a list of everyone who's made at least twenty commits as a volunteer (so including contractors outside their contract work) in the last couple of years, or whatever. Round up a smaller group of them in #mediawiki or wikitech-l and have them make a survey, with questions like: "What motivated you to volunteer for MediaWiki? What do you think would encourage more volunteers? What are the biggest things that prevent you from contributing more? Have you contributed to Wikimedia-sponsored projects like the Usability Initiative? If not, why not?" Preferably, do not allow any paid staff to have the final decision on what goes into the survey -- ask a volunteer to decide. Send the survey to the whole list, post the answers publicly, and let everyone digest them.
What has happened so far in this thread, by and large, is that paid employees have said they don't think things like private mailing lists are a problem. But if you're trying to involve volunteers, paid employees' opinions don't matter. You need to find what volunteers think. Maybe in the end you'll decide that something would annoy paid people too much to implement, or whatever -- but you have to start by acknowledging what is actually turning off volunteers, according to them, not what you think is or should be turning them off. This has not happened so far.
Aryeh Gregor writes,
I'm not assuming that -- I've been idling in the secret channel for a while now. (I keep almost saying its name, argh. Channels that aren't access-restricted and rely on secret names are annoying.) Most of it is just chitchat. But that's exactly something that the broader community should be part of, so that staff doesn't form its own group that excludes volunteers.
As a student of human communication, I'd like to affirm what Aryeh says. Chit-chat is phatic communication -- part of group-forming and the way that people form social ties and learn how to work together.
Saying "we chit-chat over here" and "discuss development over there" imagines a false dichotomy between the two kinds of conversations, and draws a line between the two groups. It ought to be "we chit-chat and discuss development together."
Pete
Aryeh: first off I want to thank you again for the constructive criticism.
I work at the WMF's SF location, and I do agree that some the problems you're talking about are serious. I do worry sometimes about the tension between centralization and community development. I don't want to be part of something that, in creating a new resource for Wikimedia, also begins to neglect the community that gave rise to it.
On 9/3/10 10:39 AM, Aryeh Gregor wrote:
The problem isn't necessarily the actual content of what we can't see, but the mere fact that so much is clearly hidden. It draws a line that need not exist.
I wasn't part of the WMF until recently, but I don't think this is a reasonable request to make.
I think you are not really appreciating that the WMF employees are also human beings. We share an office. We go out to lunch together every day. If you banned one form of private communication, others would spring up. If you eliminated the WMF's SF location, then the limiting factor would become who has the ability to go to conferences, or the people that tech leads like communicating with, and so on.
Your argument is that there should just not be any chance for such imbalances to arise. But like it or not, as long as the servers and the money are under control of a single organization, there will always be a hierarchy of who's "in the know" and who isn't. And people like yourself are going to have to remind us when we are failing to be as open as we could be. This is never going to get solved permanently, we can only keep evolving.
Frankly I think you are better off with people from the tech community actually being in the WMF offices. It's really the managers and the fundraisers who, sometimes being non-technical and having only a short history with the Foundation, are most likely to misunderstand the community, despite their best intentions.
In other words, having geeks in the office to advocate for your concerns is a bigger plus than you realize.
And starting this discussion is a great example of that. I can already see it's become more of a "top of mind" issue at the WMF. To some extent we were already thinking about these concerns (see Danese's email) but maybe this will accelerate change.
When you hide things from volunteers, you cannot turn around and blame them for inaccurate speculation about what you've hidden.
I don't see anyone deliberately hiding things.
It's more the case that we don't have established procedures about how to be open, in a regular, repeatable fashion. We try really hard but the efforts are always competing with just trying to get things done.
And in part that openness has practical limits, for exactly the same reasons that a network card has limited bandwidth.
But somehow most open-source projects do very well without them, including projects much bigger and better-funded than MediaWiki.
http://en.wikipedia.org/wiki/Survivorship_bias
You're not counting all the ones that imploded because of a lack of in-person, high-bandwidth communication with collaborators. Most open source software projects die early.
When Wikipedia started, it wasn't a perfectly distributed effort either; they always had developers and people like Jimmy Wales in the same room. I don't really know, but I would guess that this was crucial to their initial success.
Also, few people are able to make things like usability or design work in a distributed fashion. I don't know of any open source project that's made that work.
We could try harder, maybe publish some design guidelines, or figure out some way to allow remote developers to test interfaces cheaply. But I expect there will always be a need for some centralization there.
I can say that despite being a nobody at Mozilla and having gotten only one (rather trivial) patch accepted, I feel like I'm taken more seriously by most of their paid developers than by most of ours.
You are making a lot of references to the Mozilla organization (directly or obliquely) so we might as well address that.
I asked a friend of mine who works at Mozilla and he said that they have exactly the same problems that you discuss -- there are many people who regularly complain about how the organization isn't open enough. As an outgrowth of a Silicon Valley company, that relies on advertising revenue, they are far more like a traditional software organization. My friend was even working on a secret project (Rust) for months before it was announced, which would be unacceptable to Wikimedia.
That said: they obviously have the longest experience in balancing between centralized and community development, and we can learn a lot from what they do. Especially since you report you enjoyed your experience as a first-time committer.
It is my impression that the reason they are able to be so open and communicate well about roadmaps and so on is because they do have enough resources (centralized) to do such coordination and make sure to publish documents and do press releases and whatnot. If everyone was dispersed I don't think they'd be very successful at this task.
Except that volunteers can't do the work on some projects, because we're treated as outsiders and aren't interested in committing lest we be summarily reverted.
Actually, even as a paid developer, I feel the exact same way about all of MediaWiki and Wikimedia.
It is pretty difficult to contribute to anything here; the developer takes on the entire task of educating themselves about what to do, and the way people start a conversation with you is by telling you that you're an idiot / reverting your changes. This is why I've refrained from checking in a lot of things even on a project where I am pretty much the sole developer.
I also have the experience of finding "secret" IRC channels where important communication and socialization happens. I didn't know about half the IRC channels that you all use regularly until I'd been here for months.
So I agree there's a problem but I don't agree it is specifically about the WMF or the SF office. We need to be looking into realistic policies and procedures that enable everyone to trust each other better. I know that this is possible because I've worked on projects that did have problems like you state, and how they were fixed.
On 04/09/10 03:39, Aryeh Gregor wrote:
On Fri, Sep 3, 2010 at 1:20 AM, Tim Starling tstarling@wikimedia.org wrote:
Your recommendations seem insensitive and unrealistic. What works for you does not necessarily work for everyone.
It works for many, many open-source projects.
I don't think you really know that. It's hard to see how much work goes on behind closed doors when you only have a cursory involvement with the project.
None of the open source projects I've been involved with fit the model you describe. For instance, Squid makes heavy use of face-to-face meetings, despite their geographically distributed development team. PHP has projects developed by individuals and small groups which are then reviewed on the mailing list.
Does Wikimedia want to be among them, or does it want to be among the projects that are open-source but have a stagnant community because they don't involve volunteers enough and the code is tightly controlled by a sponsoring organization? There are countless projects of both types -- both strategies work, to a degree. Completely closed-source development works too. Wikimedia is free to pick whichever model best fits its needs and ideals.
I think that's a false dichotomy. For a start, control and design are two different things, as the case of the collapsed interlanguage links demostrates. Just because an individual or small group designs something doesn't mean the community will accept it.
Design by individuals or small groups, with open community review and consensus decision-making, is entirely consistent with a thriving community. I'll grant that it's not be ideal, and that an open design process should be encouraged where possible, but it's not a contradiction as you seem to imply.
My goal as a developer is to support the community such as it is, not to browbeat shy or otherwise sensitive developers into either posting their comments in public or keeping their thoughts to themselves. And my goal as a community leader is to seek consensus on all decisions and to defuse conflict wherever possible by finding compromises. I don't think these two things are contradictory.
I can say that despite being a nobody at Mozilla and having gotten only one (rather trivial) patch accepted, I feel like I'm taken more seriously by most of their paid developers than by most of ours.
I'm sorry to hear that, and I'd like to know (off list) which paid developers are making you feel that way. I've made it pretty clear to Danese that I think you're one of our best volunteer developers, and that you should be given whatever you ask for, which, I think it's understood, includes respect. Maybe I can help to get that message to filter down to the rest of the organisation.
I know more about what Mozilla is planning to do in their next release than what Wikimedia employees are planning. I've had lengthy discussions on occasion with core Mozilla developers, and sometimes I've gotten something changed because of it. I can say none of these things about any Wikimedia project that's dominated by paid employees.
There are two projects which Wikimedia employees are working on for the next core release: the resource loader and the new installer. Both of them have been discussed on wikitech-l, and both have invited community involvement by way of design documents published on mediawiki.org. Both of them did their initial development in a branch, which anyone was free to review and contribute to.
-- Tim Starling
On Thu, Sep 2, 2010 at 8:40 PM, Roan Kattouw roan.kattouw@gmail.com wrote:
- Shut down #wikimedia-dev (formerly #wikipedia_usability, kind of).
The explicit purpose of the channel is to allow development discussion with less noise, but "noise" here means community involvement. In community development, you do get a lot more discussion, but that's not something you should try avoiding. In general, use existing discussion fora wherever possible, and if you do fragment them, make sure you don't have too much of a staff-volunteer split in which fora people use.
No, "noise" means bots and people trying to support people with questions like "how do I disable anon reads on my wiki" as opposed to developers (paid and unpaid alike) being engaged in a design discussion. Maybe #wikimedia-dev should be renamed to #mediawiki-dev to remove the suggestion of WMF-exclusivity, but I definitely see the value of a channel dedicated to communication between developers with support questions and bots kept out.
As I've said elsewhere to people, this isn't an excuse for fracturing the discussion. Using a single channel for development *and* support has worked for *years* until the Usability Initiative decided it needed its own channel.
The channel is not actually that busy if you don't count the bots. And for those of you who *really* don't like them, you can /ignore them. I don't consider people asking for help "noise" either, it's part of being engaged with the community.
I don't see the need for a separate development channel at all.
-Chad
On Fri, Sep 3, 2010 at 3:05 AM, Aryeh Gregor Simetrical+wikilist@gmail.com wrote:
- Consider what to do about code review. This is pretty much the
hardest problem on this list, which is why I don't propose a specific solution here, but there has to be a better solution than "assume a bunch of employees are trusted enough to sync their own code, force everyone else to wait months for central review".
There are three ways this problem may be dealt with: * People responsible for code review focus on code review. * People responsible for code review involve more people to review code. * People responsible for code review don't do code review and don't want to lose the control over code review process -- what is done now, does not work. The review backlog is one of the things I stopped contributing at some point.
- Stop concentrating tech employees in San Francisco. Either have
most of them work from home, or perhaps establish other small offices so that they're split up. The point is, make them rely on telecommunication, because if you put people in the same office they'll talk a lot face-to-face, and volunteers simply cannot participate. The purpose of putting people together in an office is so that they work together as a team, and this is exactly what you do *not* want, because volunteers cannot be part of that team. This is the second-hardest problem, or maybe the hardest, and I can't give a full solution for it either. I'd suggest checking with Mozilla about how they do it, because I know they do have offices, but they're a perfect example of community-oriented development.
That won't work. Face-to-face communications are extremely efficient for many things (like Roan pointed out, it works well with code review).
- Explicitly encourage all paid developers to do everything in public
and to treat volunteer developers as they would paid ones. I'm not saying this should be enforced in any particular manner, but it should be clearly stated so that everyone knows how things are intended to be.
Or at least document all their decision in public.
- Shut down the secret staff IRC channel. Development discussion can
take place in #mediawiki, ops in #wikimedia-tech, other stuff in #wikimedia or whatever. If users interfere with ops' discussions sometimes in #wikimedia-tech during outages or such, set all sysadmins +v and set the channel +m as necessary. That's worked in the past.
AFAIK this is mostly a sysadmin channel.
- Shut down #wikimedia-dev (formerly #wikipedia_usability, kind of).
The explicit purpose of the channel is to allow development discussion with less noise, but "noise" here means community involvement. In community development, you do get a lot more discussion, but that's not something you should try avoiding. In general, use existing discussion fora wherever possible, and if you do fragment them, make sure you don't have too much of a staff-volunteer split in which fora people use.
I'd rather rename it to #mediawiki-dev.
- Stop using private e-mail for development, at least to any
significant extent. If there are any internal development mailing lists or aliases or whatever used for development, retire them.
Or at least make usability@wikimedia.org a publicly logged mailing list. I see no reason why it should not be (you may create a separate internal mailing list).
2010/9/3 Victor Vasiliev vasilvv@gmail.com:
Or at least make usability@wikimedia.org a publicly logged mailing list. I see no reason why it should not be (you may create a separate internal mailing list).
It's not really being used anymore because the Stanton grant is over now, and the people that used to work on it are now working on other tasks while still supporting the existing UsabilityInitiative / Vector software and performing rollouts such as the one this week. Other such project-specific mailing lists should preferably be public, yes.
Roan Kattouw (Catrope)
The quotes below are illustrative excerpts, my replies are to the whole post.
On 03/09/10 09:05, Aryeh Gregor wrote:
That's what leads to things like http://www.mediawiki.org/wiki/Special:Code/MediaWiki/67299. Some people said that maybe that could have been phrased better, or something. But the revert wasn't the problem, it was a symptom of the problem. The problem was that the design was decided on somewhere that volunteers couldn't or wouldn't participate. Of course you revert something that contradicts an agreed-upon design -- the problem is that the agreed-upon design was only agreed upon by a small group of employees. How are volunteers supposed to contribute in that environment, if they don't know what tune they're supposed to be dancing to?
The usability team has shown some amount of arrogance and aloofness in their dealings with the developer community, and I understand that you were upset by that. But I don't think arrogance is a trait which is confined to employees, in fact I think it's a near-universal fault. Being able to get stuff done in spite of it is an essential skill.
I think that all developers care about usability, and that UI design should be a part of every project team. I don't think we should have one team writing bad interfaces, and another team rewriting them to be "usable", it's inefficient. Ideally, UI experts should be available to be assigned to any project, and this is already happening to some extent. Project-based teams should hopefully be more open and accessible than the usability team was.
As for fundraising, the work is uninspiring, and I don't think we've ever managed to get volunteers interested in it regardless of how open we've been.
- Shut down the secret staff IRC channel. Development discussion can
take place in #mediawiki, ops in #wikimedia-tech, other stuff in #wikimedia or whatever. If users interfere with ops' discussions sometimes in #wikimedia-tech during outages or such, set all sysadmins +v and set the channel +m as necessary. That's worked in the past.
Your recommendations seem insensitive and unrealistic. What works for you does not necessarily work for everyone.
Some contributors (both employees and volunteers) are not comfortable talking in a public forum, and prefer to not say anything at all than to broadcast their ideas to the world on a publically logged IRC channel or mailing list. I used to rail against such sensitivities, but I've come to see that as unproductive. I still think that we should encourage people to use public forums, but only to a point, and that point should not be "use the public forum or don't contribute at all", as you seem to be suggesting.
-- Tim Starling
Thanks you Aryeh for your excellent comment (Score: 5, Insightful).
I fully agree that excluding volunteer coders from decision making processes is a dangerous path, which in the long run could cost WMF more than the time spent on including the community in a collaborative and open way.
As an occasional volunteer coder (most of my contributions to Wikimedia are on other projects), my focus is less on lack of involvement in decision making (even though I can fully understand that being an issue for more experienced volunteer coders). A lot frustration emerges from the huge code review lag, which has been a known problem for a long time and I don't see any improvement despite the large amount of hiring WMF has conducted. I for one have mostly withdrawn from my attempt to get involved more deeply in MediaWiki's development.
-- Tobias
On 2010-09-02, Aryeh Gregor wrote:
Over the last couple of years, MediaWiki development has moved from being almost entirely volunteer-based to having a large contingent of paid developers. A lot of people have noted that this has led to a lot of work being done without much community involvement. Just for a basic statistic, in July, I estimate that about 90% of non-localization commits to extensions/UsabilityInitiative/ were by paid employees. (I use "employee" loosely in this post, to include all paid staff, such as contractors.) By contrast, about 25% (ballpark figure) of non-localization commits to phase3/ were by paid employees, and the number of volunteer commits to phase3/ was much higher than the total number of commits to UsabilityInitiative, so this isn't just a matter of community members not doing as much work overall.
...
I'd just like to repeat what others have said, and note that I agree with Aryeh's comments and replies 100%.
It's very dissapointing to see many of the suggestions discarded almost immediatley by most of the staff members replying as "unrealistic". These are well thought out points and I think a decent amount of consideration should be given to all of them, regardless of anyones predispositions.
Robert
On 9/3/10 4:55 PM, Robert Leverington wrote:
It's very dissapointing to see many of the suggestions discarded almost immediatley by most of the staff members replying as "unrealistic".
I can't speak for others, but I have to say that the idea of having paid developers without a space for them to work together in person is very unrealistic.
My first month working for the WMF was as a remote contractor, and let me tell you it was a miserable experience.
There is a reason why virtually every job posting for a programmer in any organization says "no remote contractors". It is almost impossible to work effectively in a software organization without being physically in the same room at least part of the time, and it's especially difficult to ramp up quickly as a new employee.
Now, with open source projects, contributors have already self-selected, passed through many filters, and probably gone through a slow process of learning how to contribute. (But you should consider though that something like 9/10 people who want to contribute to such projects usually find the barriers too high). So, people who are already part of the Wikimedia community can probably tolerate more remote work. But even open source projects often hold "hackathons" or other coding get-togethers, and I think everyone will agree these are far more productive.
To his credit Aryeh is aware of this but he believes that the productivity hit is worth it if it ensures the organization is 100% transparent. I just don't agree we need to go to such lengths.
That said -- there are techniques which we could adopt which could help equalize things. For instance, Nat Friedman believes that when conducting conference calls, everyone should dial in (even people who could be in the same room).[1]
[1] http://nat.org/blog/2010/04/everyone-dials-in/
On 2010-09-03, Neil Kandalgaonkar wrote:
On 9/3/10 4:55 PM, Robert Leverington wrote:
It's very dissapointing to see many of the suggestions discarded almost immediatley by most of the staff members replying as "unrealistic".
I can't speak for others, but I have to say that the idea of having paid developers without a space for them to work together in person is very unrealistic.
In the past all paid developers worked remotely (at least, not in the same office as one another), and there still are paid developers who work remotely. Additionally, all volunteers work remotely. Based on my experience with MediaWiki I would say that development in the past was significantly more efficient and community involved than it is currently.
I can understand how this suggestion in particular might not be entirely realistic, but it is only one of the suggestions that have been put forward and regardless of whether it is taken as a whole has some amount of merit in the context of this discussion.
Robert
2010/9/4 Robert Leverington robert@rhl.me.uk:
In the past all paid developers worked remotely (at least, not in the same office as one another), and there still are paid developers who work remotely. Additionally, all volunteers work remotely. Based on my experience with MediaWiki I would say that development in the past was significantly more efficient and community involved than it is currently.
http://en.wikipedia.org/wiki/Post_hoc_ergo_prompter_hoc
The fact that we have scalability (in terms of code review) and transparency issues now that we have a number of devs in one office while we didn't have them back in the day when there was only one dev at the office doesn't mean the concentration of developers in the office *caused* these issues, much less that undoing said concentration will fix them.
For instance, the activity level in the MW SVN repository grew significantly about 2 years ago if memory serves [1] , and our code review infrastructure shrunk by 50% with Brion's departure just under a year ago rather than being expanded. This has to be one of the main causes of the current code review situation, and I don't believe concentrating devs in the office made much if any difference here.
Certain transparency issues that have been mentioned probably are related to having an office, but you'll still need to make sound arguments to support this notion (fortunately, some people have done this) rather than committing a logical fallacy. You can't just blame any arbitrary event that occurred in the past 5 years for everything that's worse now than it was 5 years ago without backing up that assertion with convincing arguments.
Roan Kattouw (Catrope)
[1] These numbers blow my mind every so often: when I started in July 2007 we were in the r26000s vs. the r72000s today, even though the SVN history goes back to 2001.
On 9/3/2010 11:55 PM, Roan Kattouw wrote:
2010/9/4 Robert Leverington robert@rhl.me.uk:
In the past all paid developers worked remotely (at least, not in the same office as one another), and there still are paid developers who work remotely. Additionally, all volunteers work remotely. Based on my experience with MediaWiki I would say that development in the past was significantly more efficient and community involved than it is currently.
http://en.wikipedia.org/wiki/Post_hoc_ergo_prompter_hoc
The fact that we have scalability (in terms of code review) and transparency issues now that we have a number of devs in one office while we didn't have them back in the day when there was only one dev at the office doesn't mean the concentration of developers in the office *caused* these issues, much less that undoing said concentration will fix them.
For instance, the activity level in the MW SVN repository grew significantly about 2 years ago if memory serves [1] , and our code review infrastructure shrunk by 50% with Brion's departure just under a year ago rather than being expanded. This has to be one of the main causes of the current code review situation, and I don't believe concentrating devs in the office made much if any difference here.
It grew, then has mostly been dropping since. The total number of commits is down from a peak in 2008. There were 5% fewer commits overall in 2009 than 2008, and there were 20% fewer in phase3.
4 of the top 5 months for most phase3 commits are in 2008. Based on the number of 2010 commits to date, it will be a similar drop this year (3% overall, 21% phase3)
I made a graph of phase3 commits per quarter - http://www.mediawiki.org/wiki/File:Commits.png The second quarter of this year had the fewest commits since Q3 2006.
Certain transparency issues that have been mentioned probably are related to having an office, but you'll still need to make sound arguments to support this notion (fortunately, some people have done this) rather than committing a logical fallacy. You can't just blame any arbitrary event that occurred in the past 5 years for everything that's worse now than it was 5 years ago without backing up that assertion with convincing arguments.
Roan Kattouw (Catrope)
[1] These numbers blow my mind every so often: when I started in July 2007 we were in the r26000s vs. the r72000s today, even though the SVN history goes back to 2001.
Hi Aryeh,
Thanks for bringing this topic up. It looks like its been a pretty productive conversation so far, so I hope I don't ruin it. ;-)
Here's where I think you and I are on common ground. We seem to disagree about magnitude (e.g. "more" vs "all" or "less" vs "none"), but I think we can agree on direction: a. More peer-to-peer communication needs to occur on public channels. As many of you know, prior to starting at Wikimedia Foundation, I had been a (very) peripheral member of the development community (lurking on IRC, contributing a patch or two, developing extensions, attending Wikimania 2006 dev days). I knew I had a lot to learn about the development process, but I've been surprised about how little I did know, which is in part due to the fact that many conversations that I expected to be public and discoverable by search engine weren't. It makes it much easier to involve and vet newcomers if we can watch their participation in our peer-to-peer communications. b. Volunteers need the opportunity to participate as peers. If someone demonstrates competence, a track record for useful contribution, and an ability not to make the existing core contributors all stabby, they deserve a seat at the table. We can't make everyone a full development peer in the same way that we can't make everyone a sysop in their respective editing community, but we should push the boundaries the same way our editing communities do. c. Being on the staff at Wikimedia Foundation shouldn't automatically confer any special authority on MediaWiki trunk or even an automatic seat at the table when deciding what belongs there. Of course, many of the people at WMF are there because they've earned a special place in the community already, but getting hired shouldn't be a shortcut to that. Wikimedia Foundation staff dominates MediaWiki development as the primary financial contributor to its development, and by virtue of running many of the highest traffic MediaWiki websites. But we shouldn't have norms which keep other organizations and individuals from operating as peers in a more diverse developer ecosystem. Many of the greatest open source projects have organizational diversity in their contributor base (Linux, Apache, etc), and it's great to aspire to that. d. No one should be so sure to what degree Wikimedia Foundation employees need exclusive control of the code that runs on Wikimedia's production servers. There is a certain unique responsibility that Foundation employees have to keep the servers running, and to support the strategic goals as laid out by the community, the board, and our executive staff. However, that doesn't mean the correct answer is to become control freaks or lock out the volunteers that helped us get where we are, let alone shut the door to new volunteers. It just means that, unlike with point "c" above, we're in much more uncharted territory. There aren't many examples of open source projects for which the community is as involved in the version and configuration of the software running on the production cluster as this community is.
The difference between "c" and "d" above is very important, because it seems to be where the core of the disagreements are about direction setting. If most non-Foundation developers that have weighed in on this thread are most interested in "c", this would be a straightforward problem to solve. However, I suspect many people care about "d" every bit as much, and as a result, I think we're talking past each other a little bit. I'm not going to stake out a firm position on "d"; it's a complicated matter, and I think I can argue both sides of it. I think it's a fruitful area for discussion on foundation-l, since a lot of the tension has to do with non-technical staff and community members being more involved in the direction setting now that the Foundation budget actually allows for such a thing (and some of the direction-setting comes from the budget itself; e.g. directed grants with deliverables and schedules).
Points "a" and "b" are points that I plan to push harder on, partly as a result of this thread. The monthly update that the EPM team put together [1] is a step in that direction. Merely broadcasting what we are doing isn't sufficient to actually achieve a peer-to-peer relationship, but the information is an important prerequisite. We should be able to draft the October version of the update on mediawiki.org.[2] You now should have a better idea of the program managers to pester to get involved in something we're working on. There are still plenty of internal meetings, but I think I might be able convert at least one or two of the ones I'm responsible for into public IRC-only in the near future. Inviting the general public to our telecons won't scale, but I think we should keep an open mind about inviting key non-WMF contributors to more of our conversations, and come up with more collaboration tools that scale better but also have many of the positive properties of face-to-face collaboration and/or telecons. One potential model to look at is the IETF model [3], where face-to-face conversations are acknowledged as a necessity, but decisions aren't final until they're made on the list. I suspect it will depend on the nature of the decision and the speed with which it needs to be made (the downside of emulating a standards body is that all of them, including the IETF, are legendary for being rather slow to act).
Aryeh, I doubt we'll go as far or as quickly as you probably are hoping for, but I think it's fine to push for more and keep us honest.
Rob [1] September WMF Engineering update: http://techblog.wikimedia.org/2010/09/wmf-engineering/ [2] Stub draft of October WMF Engineering update: http://www.mediawiki.org/wiki/WMF_Engineering_Overview_October_2010 [3] Tao of IETF: Section 5.2 "Getting Things Done in a Working Group": http://www.ietf.org/tao.html#getting.things.done
Well, this is probably my last post on this subject for now. I think I've made my points. Those who don't get them yet probably will continue not to get them, and those who get them but disagree probably will continue to disagree. It looks like nothing big is going to change right now, but I hope that when Danese gets up to this, we'll see real improvements and not just attempts to paper over the problem without properly understanding it.
I'll just make a few further brief points to reiterate some things I said that seem to still be misunderstood:
On Sun, Sep 5, 2010 at 10:27 PM, Tim Starling tstarling@wikimedia.org wrote:
I don't think you really know that. It's hard to see how much work goes on behind closed doors when you only have a cursory involvement with the project.
It's pretty easy to figure out that there aren't daily (or weekly or monthly) face-to-face meetings among developers who live scattered across the world.
None of the open source projects I've been involved with fit the model you describe. For instance, Squid makes heavy use of face-to-face meetings, despite their geographically distributed development team.
Just to be clear: face-to-face meetings are great, in moderation. I'm totally in favor of them. But having lots of conferences is not the same as working in an office together.
I think that's a false dichotomy.
It is. There's a spectrum of middle ground in between, but the endpoints are perfectly tenable as well. I think that, given Wikimedia's mission as well as practical concerns, moving MediaWiki development significantly further toward openness would be a good thing.
I can say that despite being a nobody at Mozilla and having gotten only one (rather trivial) patch accepted, I feel like I'm taken more seriously by most of their paid developers than by most of ours.
I'm sorry to hear that, and I'd like to know (off list) which paid developers are making you feel that way.
It would be unfair to name anyone, in public or in private. If I've had negative experiences with some paid developers, that should really count in their favor, because it means I have had *some* experience interacting with them, period. If we exclude paid developers who were preexisting community members:
* I can think of two who I see with any regularity in #mediawiki. * I can think of maybe three who I've had more than one conversation with on IRC ever. * I don't think I've ever seen a wikitech-l post from the majority of them.
I can't think why most of them should even know who I am, except now maybe some disgruntled volunteer who's making trouble for them. Why would I *expect* them to respect me?
On Tue, Sep 7, 2010 at 8:29 PM, Ryan Kaldari rkaldari@wikimedia.org wrote:
First of all, all this talk of secret listservs and IRC channels is malarkey. Yes, there are private listservs and IRC channels. All of them are private for very specific and well-established reasons. Most of them are only used in very specific circumstances (for example if there was a security breach that needed to be discussed privately) and tend to be very low traffic. They are not the places where important decisions are made.
1) Either paid developers are coordinating someplace where volunteers don't see it, or they're not coordinating at all. The latter is implausible, so it's the former. It makes no difference if it's face-to-face meetings, teleconferences, IRC, or mailing lists, or even a technically public place that volunteers don't know about -- it's hidden.
2) The secret IRC channel is not low-traffic. The 1000th line before now in #wikimedia-tech (excluding parts/joins/etc., also excluding /me for simplicity) was about five days ago:
$ grep -v '[^ ]* [^ ]* *' FreeNode-#wikimedia-tech.log | tail -n 1000 | head -n 1 100903 16:08:55 <jps> and if you are only doing those in groups of 10, you need to multiply by at least 3
Doing the same on my log of the secret channel gives 100903 00:03:40, meaning it has roughly the same traffic level as #wikimedia-tech over that period. Anyone who hangs out there can tell you that almost nothing there is secret. I can't speak for private-l, because I'm not on it.
Secondly, the idea that developers here in the office don't interact with the community is absurd. The developers here interact with the community constantly.
If the goal is to attract volunteers and make them feel part of the community, it doesn't matter whether the paid people think they're doing a good enough job. It matters whether the volunteers think it. I'm pretty sure it's clear by now that practically none of us do. As I said, anyone interested in fixing the problem would do well to start by surveying volunteers rather than looking at the issue from their own perspective, and Danese told me she does plan to do that -- so I'll wait.
wikitech-l@lists.wikimedia.org