Okay, so here's my take...
First of all I want to thank Aryeh for taking the time to write out the original mail in the thread. You've obviously done quite a bit of thinking. I realize there has been discontent and even concern with the way things have been / are between Foundation tech staff / paid contractors and "everyone else" on the volunteer dev side of this Project upon which we all toil. Aryeh, your mail was a well articulated instance of those feelings.
I'm not going to deconstruct your suggestions; I actually agree with several of them (although I think you'd find the reality of flagship open source projects somewhat different than the legend). Instead I'm going to make a more general statement about why and how we're trying to shape our future efforts...
As I came into WMF, the UX project was still in full swing and we decided not to interrupt their momentum until the Stanton Grant was finished. I focused my efforts instead on planning for the new Fiscal Year's budget and hiring plans and on observing the work patterns of the existing Tech staff, the expectations and needs of the Programs staff and the community interaction dynamic. During most of this time I've been close to silent on public lists because my opinions were still in formation. I figured there was nothing worse than somebody rushing in with a lot of uninformed blathering.
Gradually I decided these were the most serious problems (in order of importance) to tackle with a new design WMF Tech:
1. Eliminate single points of failure / bottlenecks 2. Reconfigure into teams designed to encourage faster (shorter duration) and more accurate projects / deployments 3. Develop programs to encourage / grow volunteers into paid staff...recognizing that as a non-profit WMF needs to plan for more frequent turnover in the Tech department to ensure that we can grow expertise across the dev community rather than concentrating it in a few core people. 4. Restore the balance between community-led and foundation-led efforts so WMF feels again like a partnership rather than concentric circles of permission / blame
Once I recognized these objectives, I started working out how to reach them. Notice that they are in order of importance above...starting with hiring / training / documenting and otherwise creating redundancies to mitigate the SPOF problem (in Operations to keep the sites functioning for instance) is a big focus of the hiring plan for this Fiscal Year. Fully 1/3 of the Tech hires in this fiscal year are SPOF related.
Item 2 is necessary to again make us very productive (as at least one respondent on the current thread believes we should be given there are "more of us than ever before"). Experienced developers know that more hands aren't necessarily better unless you can modularize work to form small agile teams within the larger structure to get discreet bits of work done. The Linux Kernel team does this, and Apache projects are implicitly organized this way. Most of the expense of this re-org hits in the next fiscal year (hiring engineering program managers to help teams stay undistracted and productive and to maintain an overview so discreet bits of work add up to the correct whole), so this year about 1/3 of the Tech hiring will serve this item.
But the crux of Aryeh's original email is not about the first two items on my list...it really rests in items 3 and 4.
It is my belief that the Foundation can not and should not plan to fund or achieve most development projects cathedral-style. To do so would not be consistent with our founding principles (and in any case wouldn't be possible given our resources). At the same time our Projects have grown in both complexity and popularity to the point where we can't we solely expect volunteerism to provide for all needs.
There has to be a partnership...not a detente. To that end it is clear we need to improve transparency and afford more opportunities for participation and cooperation between paid Tech staff and the volunteer developer community. We need to undertake this shift with realistic expectations. We are never going to be interested in every scrap of volunteer engineering ever undertaken, but we are going to take steps to make it easier for volunteer engineers wishing to contribute to figure out what contributions are sought and how to gain increased responsibility in our essentially meritocratic community (which is how Mozilla finds their new paid staff as well). We're going to create more entry points and to get volunteers more involved. In this fiscal year 1/3 of the new Tech hires are in service of items 3 and 4...
If you're still reading, then you must really care to know what I think. Thank you for that. Mostly I wanted to share that re-balancing the volunteer-to-paid staff dynamic is definitely a first-year priority for me, although its not more urgent than establishing a new primary Data Center and making sure we have staff to keep the site running...but its coming. We're going to co-create (or re-create it). Like all work of digital "intentional communities"[1], its going to take work on all sides to get it right...and I hope you'll help.
Danese Cooper CTO, Wikimedia Foundation danese@wikimedia.org
On 3 September 2010 00:51, Danese Cooper dcooper@wikimedia.org wrote:
- Eliminate single points of failure / bottlenecks
- Reconfigure into teams designed to encourage faster (shorter
duration) and more accurate projects / deployments 3. Develop programs to encourage / grow volunteers into paid staff...recognizing that as a non-profit WMF needs to plan for more frequent turnover in the Tech department to ensure that we can grow expertise across the dev community rather than concentrating it in a few core people. 4. Restore the balance between community-led and foundation-led efforts so WMF feels again like a partnership rather than concentric circles of permission / blame
I really like item 2. When I was working on MediaWiki related stuff, the main problem I had was working out who to talk to. The answer, if you ask, seems always to be "Tim" — which is not very scalable (brilliant though Tim is). I'd hope that along with an organisation into more formal teams would come a structure where individuals are responsible (either permanently, or on rotation like the chromium) for subsets of MediaWiki/Wikimedia.
This would imply that the structure used by the Usability Initiative is something we want to emulate — a few tight-knit teams that can focus on specific concerns, containing people with the power to say "yes" or "no" to particular features/ideas. Having a person responsible for saying "no" is essential; as Danese says, you can't accept every random patch or idea that gets thrown your way, and leaving things languishing forever is less kind than just saying no (ideally with a reason). Hiding behind team decisions is impersonal to the point of rudeness — if I'm committing into your area of interest, I am part of your team.
The other advantage of having teams is that it makes it more explicit which areas of development are being focussed on, and by whom. Wikimedia obviously concentrates a reasonable amount on fundraising, which is essential as a means, but it doesn't achieve anything directly. I'd hope that having some kind of explicit structure would expose any less obvious blind-spots — my personal gripe is that no time gets spent on Wiktionary (cf. bug 20246).
Clearly there are downsides, cliques are likely to form. I'd certainly regard it as a failure of the system if it stopped someone from doing something merely because they were currently on the wrong "team" — there's not much point in keeping lists of team members, perhaps all that is needed is a team leader (responsible for accepting or refusing changes/ideas), and the team is simple those who are talking to him. In case it's not obvious, I would not make team leaders exclusively employees, but use the meritocracy previously described that would only make it likely that most of them would be.
The other sentiment I very much agree with is that more communication should be public — for the simple reason that if I don't know what you're doing, I can't help. Keeping track (however loosely) of what's being worked on is much more efficient than trying to second guess. If the channels we have are too noisy, it's easy to split them by topic. I like the idea of making #mediawiki the support channel, and having #mediawiki-dev for development — this is a model used by lots of projects on freenode. We could even have #mediawiki-offtopic too, if people want to do a lot of misc chatting. Being able to talk to other developers is very useful for getting things done, whereas having some developers who can't be contacted at all by others is very troublesome indeed. I had assumed that it was a requirement for a wikimedia staff member to be somewhere public on freenode — clearly I was mistaken.
I hope that items 1, 3 and 4 will become much easier if we solve 2 correctly, and communications will always be improvable. Good luck with your plans, and please keep us updated — I'd much rather have to add you to my spam filter than to not know what's happening.
Conrad
Στις 03-09-2010, ημέρα Παρ, και ώρα 05:34 -0700, ο/η Conrad Irwin έγραψε:
On 3 September 2010 00:51, Danese Cooper dcooper@wikimedia.org wrote:
- Eliminate single points of failure / bottlenecks
- Reconfigure into teams designed to encourage faster (shorter
duration) and more accurate projects / deployments 3. Develop programs to encourage / grow volunteers into paid staff...recognizing that as a non-profit WMF needs to plan for more frequent turnover in the Tech department to ensure that we can grow expertise across the dev community rather than concentrating it in a few core people. 4. Restore the balance between community-led and foundation-led efforts so WMF feels again like a partnership rather than concentric circles of permission / blame
I really like item 2. When I was working on MediaWiki related stuff, the main problem I had was working out who to talk to. The answer, if you ask, seems always to be "Tim" — which is not very scalable (brilliant though Tim is). I'd hope that along with an organisation into more formal teams would come a structure where individuals are responsible (either permanently, or on rotation like the chromium) for subsets of MediaWiki/Wikimedia.
I think of the "who to talk to" issue (which in part reduces to a "who can officially sign off on your code so it may be deployed" issue) as #1, bottlenecks. Over the past few years that I have been involved in various ways in the Wikimedia projects, I have seen people give up after waiting a couple of years for their code to make to the deployable stage. This in term impacts both #3 (getting/attracting volunteers who can later turn into paid staff) and #4 (dynamics between WMF and the community, to the extent that WMF staff are not considered community members.
It sort of puts a new spin on the old "If you want it done, write it yourself because we are too busy" line. When writing it yourself (and being willing to revise it until it's considered good) isn't available as an option any more, an open source project is rather less open.
I know that is by no means the only bottleneck we have; it's just the one that looms large in my mind at the moment.
This would imply that the structure used by the Usability Initiative is something we want to emulate — a few tight-knit teams that can focus on specific concerns, containing people with the power to say "yes" or "no" to particular features/ideas. Having a person responsible for saying "no" is essential; as Danese says, you can't accept every random patch or idea that gets thrown your way, and leaving things languishing forever is less kind than just saying no (ideally with a reason). Hiding behind team decisions is impersonal to the point of rudeness — if I'm committing into your area of interest, I am part of your team.
Having teams responsible (and with authority) for different areas, able to approve features and code, sounds fabulous. I know we are moving towards broader team structure already. Ideally there might be more than one person on each team who could give the thumbs up/down.
The other advantage of having teams is that it makes it more explicit which areas of development are being focussed on, and by whom. Wikimedia obviously concentrates a reasonable amount on fundraising, which is essential as a means, but it doesn't achieve anything directly. I'd hope that having some kind of explicit structure would expose any less obvious blind-spots — my personal gripe is that no time gets spent on Wiktionary (cf. bug 20246).
Clearly there are downsides, cliques are likely to form. I'd certainly regard it as a failure of the system if it stopped someone from doing something merely because they were currently on the wrong "team" — there's not much point in keeping lists of team members, perhaps all that is needed is a team leader (responsible for accepting or refusing changes/ideas), and the team is simple those who are talking to him. In case it's not obvious, I would not make team leaders exclusively employees, but use the meritocracy previously described that would only make it likely that most of them would be.
The other sentiment I very much agree with is that more communication should be public — for the simple reason that if I don't know what you're doing, I can't help. Keeping track (however loosely) of what's being worked on is much more efficient than trying to second guess. If the channels we have are too noisy, it's easy to split them by topic. I like the idea of making #mediawiki the support channel, and having #mediawiki-dev for development — this is a model used by lots of projects on freenode. We could even have #mediawiki-offtopic too, if people want to do a lot of misc chatting. Being able to talk to other developers is very useful for getting things done, whereas having some developers who can't be contacted at all by others is very troublesome indeed. I had assumed that it was a requirement for a wikimedia staff member to be somewhere public on freenode — clearly I was mistaken.
I'm one of those who posts almost never to this list (in fact almost never to any list, I have an allergy :-P). However I live in IRC in public channels where anyone can and does find me. Between those two sets of tools, I would hope that everyone could be similarly available and that their development processes would be accessible for public participation.
A brief comment about closed mailing lists, since this has come up a few times:
We have a mailing list that is typically used for operations matters. I'm not sure how much of that *has* to be private but I am sure some of it does: things like Sun contract numbers and invoices don't belong on a public mailing list. If we were to close it, we would simply have to find another way (less convenient I suppose) to communicate that same information.
AFAIK we don't have a s3krit mailing list for development. (Now someone will prove me wrong by revealing the existence of wmfdevcabal-l. :-P)
I hope that items 1, 3 and 4 will become much easier if we solve 2 correctly, and communications will always be improvable. Good luck with your plans, and please keep us updated — I'd much rather have to add you to my spam filter than to not know what's happening.
We hope to produce enough quality spam to keep you and everyone else happy :-)
Ariel
Conrad
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
wikitech-l@lists.wikimedia.org