[Foundation-l] Community, collaboration, and cognitive biases

Tue Jun 8 22:31:32 UTC 2010

Hello all,

the massive thread regarding the default sidebar language link
expansion state has surfaced a number of fundamental and significant
questions regarding the working relationship between the Wikimedia
Foundation and the larger Wikimedia volunteer community. This message
represents my snapshot-in-time thinking about these questions, and I
hope others will join this thread to meaningfully advance this IMO
very important conversation.

Tl;dr version: I believe that the transparency of Wikimedia's
engineering processes, and the general quality of these processes, has
significantly improved over the last year. At the same time, I agree
with those who are observing a widening gap between staff and
volunteer contributions, and I think we need to work together to close
this gap in full awareness of the cognitive biases present in all of
us.

1) Increase in quality and transparency of our engineering processes:

I'm very grateful for the hard work accomplished by the User
Experience Team to-date. When we started the usability project, we had
never run a discrete engineering effort of comparable scale before,
and indeed, most of our larger-scale engineering projects tended to be
drowned by day-to-day emergencies and bug fixes. The team consists
primarily of people who didn't have a long prior history in Wikimedia
projects, nor with each other: they had to grow as a team, understand
our enormously complex community, while also delivering results. I
want to use this opportunity to explicitly thank Brion, who played a
key role in the orientation of the team.

The UX team has implemented a number of important practices:

* very frequent sharing of detailed work product, including mock-ups,
specs, and research results such as user test videos.
* systematic QA using an external QA firm, browser compatibility
matrix, etc.; beginnings of automated testing
* qualitative remote and in-person user testing to assess real-world
experience with the site
* moving towards data-driven decision-making using tools such as
click-tracking, systematic assessment of feedback data, various
statistics, etc.
* public and frequent blog updates both in the regular and in the
technical blog, use of the sitenotice to announce deployments
* public sandboxes demonstrating all bleeding edge improvements early
* various community-oriented discussion pages and documentation on
usability.wikimedia.org informing and supporting design and
localization
* largest systematic feedback collection about any software deployment
in the history of Wikimedia projects

Many of these were "firsts" for the Wikimedia Foundation, and they all
were firsts on this scale. Some of these practices have clearly
contributed to letting the community participate _more_ in the overall
initiative: the larger-scale outreach, the sharing of research
findings (which led to some community-driven improvements), the
carefully prepared public testbeds, etc. Some, on the other hand, have
arguably widened the staff and volunteer gap, both between staff
developers and volunteer developers, and between staff and larger
volunteer community.

I don't think Wikimedia's development processes are anywhere near
those of proprietary web companies, but it's absolutely true that
they're also not highly comparable to 100% volunteer-driven open
source projects anymore. I think there are some parallels with both
the strengths and weaknesses of open source projects with reasonably
well-funded organizations behind them, both for-profit and non-profit.

I also want to be clear that there's no doubt in my mind that our
ability to execute projects of this _type_ and at this _scale_ is
unprecedented, and a historic achievement for the Wikimedia movement.
MonoBook was implemented in 2004, and the basics of the user interface
and user experience of Wikipedia have changed very little between then
and pre-Vector.

The reasons for that are varied. Fundamentally, I believe that online
volunteer mass collaboration works best when incrementalism and
self-selection can, over a long period of time, add to an overall
product that's useful at any given time -- such as Wikipedia or
Wikimedia Commons. We've seen this work less well where we need to
collaborate in a short period of time to achieve a result of
predictably high quality (Wikinews), or where we're working over long
periods of time on a result with (perceived or actual) limited
usefulness until it is complete (Wikibooks).

I'm not saying this is inherent in volunteer mass collaboration, or
that these challenges are insurmountable, just that a) tools and
practices of volunteer collaboration tend to support self-selected
long-term incrementalism better than, shall we say, strategic
megaprojects, b) we're not necessarily even using all the tools and
practices yet that other volunteer projects successfully employ to
tackle strategic challenges.

2) Widening gap between staff and volunteer contributions:

As the Wikimedia Foundation began to professionalize, it initially
moved its focus away from some of the early experiments in volunteer
mobilization (such as the 2006 Wikimedia Foundation committees), and
moved instead towards employing and tasking WMF staff for each
critical work area. Much of 2008 was spent for this new team to get
its feet on the ground and get oriented in the world of Wikimedia, to
the detriment of developing new structures for volunteer engagement.

The strategic planning process launched in 2009 was, from the
beginning, conceptualized to mobilize the Wikimedia community in
collecting ideas and planning its own future, and has achieved
unprecedented volunteer participation in doing so. We're now shifting
towards using the StrategyWiki to help volunteers in efforts to
self-organize around particular issues, whether the WMF is giving
attention to them or not -- we'll be posting more about that soon, and
I'm pretty excited about it.

At the same time, when tackling large scale strategic projects like
the usability initiative or the bookshelf, I do believe we haven't yet
succeeded at forming the most productive collaborative relationship
possible with the larger volunteer community. I should note that we
haven't even fully succeeded at closing the gap between staff and
contractors based in San Francisco, and those who are not -- we have a
long way to go.

The staff/volunteer gap is evident when looking at the commit history
of the UsabilityInitiative codebase, for example, which is very much
dominated by WMF staff and contractors. (The notable exception, as for
all our code, is localization -- again a great example for
self-selected incrementalism.) It's also evident in the recent
conflict between a staff and a volunteer developer regarding the
language link issue, and in our overall conversation about that same
issue.

3) Closing the gap and overcoming cognitive biases:

I do think it's possible and necessary to close that gap. I think
doing so has to _start_ with discussion, but ultimately to me this
isn't so much about conversation, but about actual real-world
collaboration with practical results -- we want to get stuff done
together, and we're all working towards the same vision.

To do so, I think we have to be wary of our respective cognitive
biases and operate with the highest possible degree of self-awareness.
I find the framework of cognitive bias personally very helpful when
I assess my own actions, and the list on Wikipedia
( http://en.wikipedia.org/wiki/List_of_cognitive_biases ) is a good
starting point. I believe we've seen some examples of biases listed
there in both WMF staff and volunteer actions, such as:

* belief that the group you're a member of is diverse, whereas other
groups are not ("the staff does/is X", "the community doesn't have
the expertise to do Y") ;
* preferential treatment of people perceived to be members of
the same group, and the associated us vs. them dynamic;
* irrational escalation (tendency to defend an investment based on
cumulative prior commitment) and the bandwagon effect
(tendency to join the belief of others).

These and other biases contribute to unhealthy spirals of escalation.
There's a big picture of clear alignment of values and interests:
we've all committed ourselves to a very important mission, and we
have seen time and again that we can work together immensely
successfully in doing so. Fundamentally, I think it's the group
division that's the most important to overcome -- inside
the Wikimedia movement, but also towards readers and "outsiders."
WMF absolutely has been both part of the problem, and part of
efforts to solve it.

I entirely agree that we need to steer clear of unhealthy power dynamics.
Wikipedia editing is not a tyranny of the majority or the minority, and nor is
our engineering and design process -- ideally, the best arguments, the
best code and the best edits should prevail. I'm glad that the BOLD,
revert, discuss cycle --
http://en.wikipedia.org/wiki/Wikipedia:BOLD,_revert,_discuss_cycle --
was invoked in prior threads, which I think is a generally sensible
framework through which we get stuff done. It's not the only framework,
but it's one that we (should) stick to wherever reasonably possible.

To the extent that there is an irrational status quo bias, I think the best
way to overcome it is for us to engage the largest possible number of
people in bringing about and advocating on behalf of positive changes.

With all this in mind, here are just a few concrete ideas for closing the gap:

1) Embedding teams funded by WMF into larger, publicly visible
workgroups which include volunteers and which meet regularly e.g. via
IRC;
1 a) Outreach to grow and strengthen those workgroups with the best
skills present in the Wikimedia volunteer community;
2) Publication of mini-projects which we identify and which can be
tackled by self-organized volunteers, with mentoring by experienced
WMF staff and volunteers (happening a bit via GSoC, but not as much
outside of it yet);
3) Making development roadmaps fully transparent and open to
discussion, and sharing justifications for all key priorities;
4) Further iteration of tools and processes for rapid volunteer-driven
bug reporting, cross-browser testing, and submission of simple
patches;
5) Stimulating larger scale contests focusing on specific areas of interest;
6) More topic-focused meetings and sprints like the multimedia
usability meeting in Paris.
7) Further experimentation with tools like IdeaTorrent for large-scale
brainstorming and ranking purposes (we have a prototype running at
http://prototype.wikimedia.org/en-idea/ideatorrent/ ).

The Wikimedia Foundation regards these as crucial questions, and
we've had many conversations about them, both internally and
publicly. I hope these conversations will continue to broaden.
I personally intend to work especially on 3) in the list above, and want
to help establish 1) as a normal practice in key WMF program activities.
Howie has started a general brainstorming page here, where I've
reposted the above:

http://usability.wikimedia.org/wiki/Product_Development_Process_Ideas

I hope you'll join me in adding to that list, and that we continue to
hold each other to the idea of building true and genuinely open forms
of collaboration, which is one of the most important cornerstones of
the Wikimedia movement.

All best,
Erik
-- 
Erik Möller
Deputy Director, Wikimedia Foundation

Support Free Knowledge: http://wikimediafoundation.org/wiki/Donate