[Foundation-l] hiding interlanguage links by default is a Bad Idea, part 2

[Gregory Maxwell]

On Mon, Jun 7, 2010 at 11:52 AM, Eugene Eric Kim <eekim at blueoxen.com> wrote:
> On Sat, Jun 5, 2010 at 1:00 PM, Gregory Maxwell <gmaxwell at gmail.com> wrote:
>> On Sat, Jun 5, 2010 at 2:03 PM,  <susanpgardner at gmail.com> wrote:
>>> Sorry for top-posting.
>>>
>>> Austin, think about who "everyone" is.  The folks here on foundation-l are not representative of readers.  The job of the user experience team is to try to balance all readers' needs, which is not easy, and will sometimes involve making decisions that not everyone agrees with. People here have given some useful input, but I think it's far from obvious that the user experience team has made a "mistake.". (I'm not really intending to weigh in on this particular issue -- I'm speaking generally.)
>>
>> Sue, you appear to be making the assumption that the folks here are
>> writing from a position of their personal preferences while the
>> usability team is working on the behalf of the best interests of the
>> project.
>>
>> I don't believe this comparison to be accurate.
>
> I agree that this comparison is inaccurate, but I disagree that this
> was Sue's assumption. All she said was that vocal outcry on a mailing
> list should not be construed as community consensus. I hope that no
> one disagrees with this.

"All she said", no. She didn't state that. You're putting words in her mouth.

Perhaps I'm guilty of the same crime. But What Sue said was

"The folks here on foundation-l are not representative of readers.
The job of the user experience team is to try to balance all readers'
needs, which is not easy, and will sometimes involve making decisions
that not everyone agrees with."

I read that as contrasting the purposes of the UX team and the people
commenting here.  I am unable to determine any other reason for
bringing these two statements together except for the purpose of
drawing the comparison I suggested was being made, even with your
proposed alternative.

We all communicate unclearly at times, — and I am more than willing to
accept that I saw a comparison there which was not intended.

In the interest of good communication I hope that you will take the
time to consider how I could have come to the understanding that I
did.   I'm sure many other people on this list had the same
understanding.

What we have here is a nearly unanimous response with respect to the
disposition of the interwiki links. If you'd like me to bring this to
the larger community I can do so but my understanding was that the
normal community process was already quashed with respect to this
change.

While no single forum is indicative of a consensus of the entire
community, the broadness of the response here is a strong indicator.


[snip]
> The bad thing is the us versus them tone in this and other messages.
> There is a larger question about ownership and decision-making that is
> subtle and hard, and we need to continue to work these out. Since I'm
> going to put myself in the vocal minority and disagree with most of
> the points in this message, I'll start with what I agree with. :-)

I don't believe that there is anything particularly subtle here.  We
have many community processes in which foundation staff are welcome to
contribute to as peers with a common interest.

When you fail to do so you have created the "us" vs "them" by your own actions.

Rather then trying to draw "us" vs "them" lines in the sand, I am in
fact pleading that the foundation discontinue doing so.

In order to do that I must first acknowledge the division which I
believe has already formed. [more on this later]

>> I think the people here are speaking up for the sake of the readers,
>> and for the sake of preserving the best of the existing design
>> principles used on the site.  I know I am.
>
> Absolutely. Assume good faith. I know you and many others feel the
> same way about the UX team

Absolutely.

I would suggest that the broader community (and not necessarily the
participants here) has greater experience than the usability team, and
even the portion of the community represented here has a more diverse
composition than the UX team.

However, if we were to combine the two— we would have something
strictly superior to the component parts.  Unfortunately, we're still
able to speak about the community and the UX teams as distinct
entities.  This division will continue so long as the relationship is
viewed in the context of "decision"/"feedback" rather than as a
dialogue between peers.


> That said, keep in mind that most people assume that everyone thinks
> like they do. This is not an us versus them thing; this is a natural,
> human thing. I heard a great tip from a psychologist once: If you want
> to know what people truly think, ask them what they think other people
> think.
[snip]


While we are on the topic of lessons in human nature, please allow me
to introduce the list to the fundamental attribution error:
http://en.wikipedia.org/wiki/Fundamental_attribution_error

We all may find it informative.


> Given this quirk of human nature, we need more rigorous ways of making
> decisions than polling people, especially small, self-selecting
> groups. Being data-driven is one of those ways. But being data-driven
> is hard, too, because you still have to interpret the data. Which
> brings me to...
>
>> I was alarmed when I heard the click rates: 1%.  That's an enormous
>> number of clicks, considerably higher than I expected with the large
>> number of things available for folks to click on.
>
> Agreed. 1% is absolutely a large number of clicks, especially given
> our overall traffic. Someone in a different message also pointed out
> that click rates alone don't tell the whole story; if they did, then
> one could argue that we should eliminate the Edit button.
>
> Good design isn't just about following the user path; it's also about
> guiding the users in a way that's appropriate to the mission of the
> work. In that vein, I think the substance of most of the messages in
> this thread have been very positive. It's resulted in data (such as
> Max's numbers) that have helped to round out everyone's understanding
> of the issue, but most importantly, it's resulted in critical context
> as to why people may be acting a certain way and why these things
> matter in the first place.


I do not believe that I am understanding the point you are making here.


The group collected on this list appears to generally hold the view
that we the interwiki links are something we should be emphasizing
beyond the level justified by their usage.

You agree that 1% is a rather large amount of usage, as Tim pointed
out— it would compare favourably to the edit feature.

>From these facts that we appear to agree on, if we were design from
the perspective of promoting values we would continue to keep the
links prominent, and if we were to be data-driven about usage we would
also keep the links prominent.

I'm not seeing anything in your above statement to suggest why would
decide not to make the links prominent.

If you're not — then why is the discussion continuing?  "The usability
team made a mistake, thanks are owed to X,Y,Z for catching an fixing
it. Sorry we stood in the way of the fixes and implied that you
couldn't, that won't happen again." Would be a fine conclusion.

'We will now factor in your critical context and emit new fully formed
sausage soon' would not be.


[snip]
> I mostly disagree with this. First, let me refer back to Sue's
> original point. The people on this list are not representative of the
> community of contributors. It is a self-selecting sample. That doesn't
> invalidate the substance of what's said on this list, but it does
> raise questions about claims of consensus when there seems to be a
> large, vocal group of people agreeing on a point. Passionate discourse
> on this list is a data point, but it's only one data point, and it
> needs to be interpreted in the context of many data points.

My prior offer to bring in a larger part of the community still
stands, if you really believe that to be an issue here.

In particular, the list of people agreeing on this point includes a
number of parties which seldom agree with each other. I have found
that to usually be an indication of an issue which will have a fairly
broad support in the larger community.

I didn't think bringing on a hundred person pile-on would help— if
anything I could see it only increasing distrust of the foundation in
the larger community.   It would leave me in a position of having to
continue to damn the foundation's actions on this and other recent
issues when I'd rather just move forward...


> Second, most people -- both staff and people on this list -- _are_
> disconnected from the needs of the majority of the readers. None of us
> represent the average reader, and so we need other data points to
> truly understand what the average reader experiences. And frankly,
> only a few people in this thread seem to acknowledge this.

The very first message on this subject was requesting additional data.

I think we'd all be interested in seeing more data on the subject.

Though I don't think it's valuable to define the meaning of the data
only after you've seen it. That just leads to circular justification,
the criteria needs to come first.

Nor do I think we need more data in order to conclude that the
promotion of the interwiki links is an important _value_ for us, and
that we wish to promote it above and beyond the level justified on a
pure ease of use basis.

Accordingly, I think the data which would be most helpful would be
indicators that were suggesting that the existence of the exposed
interwikis creates a non-trivial harm to the usability of the site,
since that is the kind of evidence which would overrule the value
based decision to promote the links.

That kind of outcome would be fairly surprising to me, and I think
everyone else here as no one has been suggesting that such an outcome
is expected.

> You're essentially claiming that certain staff members are copping a
> "we know better" attitude. That may be true (although I honestly don't
> see it in this thread), and if it's happening, it's wrong. That said,
> people on this list are copping the exact same attitude. Let's not
> draw generalities about us versus them based on a few individuals.
> Let's instead find smarter ways to make decisions that will help us
> fulfill our mission. You have been one of the best at doing this, and
> we need more of it.

I think that drawing an equivalence between comments by the staff and
community here is an error.

I expect all of us, staff and community alike, to read the comments of
others, consult their best available information, and form reasoned
viewpoints which they then present and defend vigorously but politely.
... and that as more information and more viewpoints become available
that people will have open minds to revising their positions.   This
is how discussion works.

Nothing about presenting a strong argument should be construed as a
fault-worthy "we know better".  Even in my strongest statements I am
willing to believe that I could be wrong, and I know the same to be
true of many others here—   It would be hard to sanely hold any other
position in the presence of so many experienced and intelligent
people.  I apologise for sometimes allowing it to sound otherwise.

But this discussion process only works when the participants are peers.

In this case, in spite of a mostly lacking defence and the obvious
super-majority holding the counter position, the site continued to
display the collapsed interwikis for a prolonged duration.

It is not vigorously defended positions which I am characterizing as
"we know better",  it is the obvious _absence_ of those positions
combined with the reality of the website (at the time I wrote the
message). And this characterization is further strengthened by the use
of language like "feedback" and the constant reminders of
non-representativeness as a fault of the assembled community when the
same could equally be applied to the UX team and the staff as a whole.

And once we have been divided into groups, those with the authority to
impose without broad agreement or even solid justification, and those
without... then I must answer that implicit statement of superior
knowledge with the correction: Of the two, it is the community which
holds the stronger claim.

> What's the success rate for community-run web sites?
> Drawing a company vs community-run comparison like this isn't just
> wrong, it's pointless. The level of success of all of the top web
> sites is a fluke to some extent. There are great, great community
> sites -- both grassroots and company-run -- that will never reach the
> scale of Wikimedia or Google.

The success rate for Wikipedia is one for one. It exists, it is successful.

We have a successful modality... but it is one which many people would
be surprised that it works at all.

Some people might think it reasonable to try to direct things towards
a more traditional mode of operation— with a reasonable eye towards
fixing the things we do poorly, but that would ignore the enormous
number of things our mode of operation does shockingly well such as
existing at all.

So, my points was that traditional organizations are fantastically
unsuccessful at accomplishing what we have accomplished and so there
is no reason to believe a change would be an improvement and many
reasons to believe it would be an enormous failure.

> We're facing huge challenges right now. We need everyone to help. The
> Foundation staff understands this. Most people here understand this.
> The question is, what's the best way to grapple these challenges?
>
> For starters, we need to come to a better, collective understanding of
> what it means to come to consensus.

You sound as though you believe that to be a new issue, but it isn't.
I see no reason to conclude that the challenge we face now are
materially different from the ones community has been grappling with
for many years... which has resulted in a multitude of more or less
workable compromises.

> It's not reasonable to suggest
> that every single design decision first be tested on foundation-l,

This isn't how our communities usually work in any case.

Bold. Revert. Discuss.  is a common modality.

I think that the group assembled here would largely agree that it
would have been acceptable for the UX team to make the change— even
with little to no public discussion, then not interfere with the
community to reverting it when non-trivial objections were raised,
then engage in a discussion about the ultimate disposition of the
feature.

This pattern allows a significant majority of changes to happen
without significant conflict and without the impediment of excessive
discussion.

For more critical changes a better process is

Discuss Poll Discuss [ Bold. Revert. Discuss. ] [Bold. Discuss.
Revise.]  Agree.

Iterating the sections in brackets. We've used processes like this
before for other user interface changes.

Through this process we've managed to make non-trivial user interface
changes without leaving popular clients out in the cold for weeks at a
time.

Sometimes an inclusive process simply takes more time.  But the only
deadlines we're facing are the ones we impose on ourselves. We can
afford some things being slower so long as progress is made, it's a
reasonable price to pay.


> nor
> would that lead to the best decisions, even if it were reasonable.

Best is the often enemy of good.

The decisions we make are so highly dimensional that the prospect of
ever finding a "best" decision is a laughable joke.

Even if we agreed on a definition of best-ness the chances of actually
finding the best solution to anything is infinitesimal.   Usually a
definition of bestness can't be found because everyone weighs
different values differently.

Moreover, a locally best solution isn't desirable.   I hope and expect
Wikipedia to be around 100 years from now. I hope that the decisions
we make today are the ones with the greatest return over that time
horizon (or longer).  Since no one can know what the future will ask
of us, this is just more evidence that a best decision is not possible
no matter who makes it.

I fully admit to being clueless. I object to people claiming that have
a best solution when no one can.

I think what we generally strive for is  "the best long term outcome
which doesn't suck too bad in any particular way in the short term".

In any case, Wikipedia is not a good place for people who do not see
the value in the inclusive decision and who are too concerned about
getting what they believe to be the best result.

In terms of content editing, people who are very concerned with saying
the "right" thing are usually the ones who end up blocked.

> Developers (including non-staff) need permission to Be Bold. We all

We agree.

> need to have permission to try things, to make mistakes, and to learn
> from them. If you want to draw any lessons from the top web sites on
> how to grow a successful site, that would be the most important one.

But being bold necessarily implies the entire community process.

If a change is imposed without the rest of the process, that's not
"Being bold", thats just simply a forced change.

Sometimes forced changes are reasonable- for example, a change to keep
the site operating or a reversion of a user interface change that was
blocking access to a large swath of users, but don't confuse an out of
process emergency action with the bold actions of the normal process.

> It's not clear to me that Trevor should have reverted Aryeh's change.
> But isn't it remarkable that Aryeh had the power to revert in the
> first place?

No. This is how we work here, and it's generally how we've worked here
for a almost a decade. For many of our contributors Wikipedia and this
governance has existed their entire adult life, and it is only way of
operating a large website that they've ever participated in.

I think it's remarkable that we've given people the authority to make
changes who still find our process surprising or remarkable.

It's only remarkable to people who don't appreciate the historical
context... those who view the situation as an organization being
gracious enough to let its "users" modify important parts of the site.

Allow me an alternative:  what we have is a community of people who
have contributed millions of man hours of work towards a common goal,
and in the furtherance of that goal we created an office with a staff
to handle routine administrative work and the interoperation with
external entities (as it's hard to deal with the borg-of-wikipedia
without becoming one).  We were able to do this because the public
supports our goals and the efforts of the community were successful at
meeting an important need of the public.

The remarkable thing is the payment you receive. Not the ability of
the community to make changes to the site.

... and please don't take that wrong:  I am _overjoyed_ that the
Foundation has been able to employ so many thoughtful and productive
people, but I think it is amazing that it is able to do so on the thin
slice of the value that our collective project has produced which
comes back in the form of donations.


> And that the end result was that Roan re-instated the
> revert? If the Foundation were on some power trip, would that have
> been possible?

Power trip isn't something that I said, and I don't think it
accurately expresses what I would have alleged.

I think there is a misunderstanding about authority here, not a
power-trip.  The foundation was never created to "govern" the
community.  It was created to keep the machines well oiled on the
communities behalf, sometimes oiling the machine requires a the
execution of authority, sometimes you have to take the elevator out of
service in order to improve it, but usually not and usually not much.
 Without the communities' continued governance the sites could not run
at all.  The WMF staff is far too small to even dream of replacing the
communities role in governance.

None the less, you need to look no further than the facebook privacy
policy reversal a year ago for a counter example.  Controlling and
commanding organizations routinely respond to public outcry.

That it took many days to get that far is a perfect example of the
foundation staff not acting as an equal peer with the community.

> We have this beautiful opportunity to learn how to do good user
> experience collectively and at scale. No one has figured out how to do
> this yet, but everyone is trying. There are going to be bumps along
> the way. We have to continue to Assume Good Faith and encourage each
> other to Be Bold.

So, you're suggesting that Wikipedia became a top 10 website, reliably
outperforming commercial clones like answers.com that had professional
development staffs, all our content, and then some,  with an acutely
_bad_ user experience?

No?

I think we _had_ a beautiful opportunity to take an already effective
and innovative development system and take it to the next level,  but
instead we tried to supplant it with the same cathedral development
model plus 'feedback' which is used by most commercial web-properties.

It wasn't clear to me that what was happening since a lot of community
development appears almost fully formed from a single creator or team,
 the distinction only became clear when the response to concerns was
"accepting feedback" rather than cooperation.

The fact that my blackberry _still_ can't load Wikipedia after weeks
is clear evidence that a change in how things are run has happened.

I took the time to respond to this thread, and to your message,
because I'd like to clear up the confusion and restore the opportunity
for cooperation as peers while it is still early enough for such a
mild remedy.

>> Somehow, the community knows how to take the ragtag assembly of its
[snip]
>> with broad appeal and generally consistent, if somewhat strange,
>> performance.
>
> In some areas yes, in others, no. We all still have a lot to learn,
> and we need to learn this together.

Absolutely, the community needs to learn how to address all areas.
We've been learning how to address all areas for a long time.  One
thing we learned a long time ago is that asserting hard-authority over
the community generally generates more heat than light, that
cooperation as equals is usually more productive, and that other means
should only be invoked as a last resort.

This isn't the same as accepting the notion of a staff that learns how
to accept feedback, and a community that needs to learn to only whine
and hope for change instead of taking action. ... the impression that
I and others have received here.

>> I think it's unfortunate that the foundation has an apparent
>> difficulty in _contributing_ without _commanding_.
>
> I don't see evidence of that here.
[snip]
> I agree with this. Again, I don't see evidence of exclusion happening
> right now. I think the discourse has been positive, I think there are
> still some things that need to be worked out, and I think the right
> thing will happen in the end.

I think your inability to see what is obvious to me and many others is
an example of the problem.

I'm confident that the right thing will happen in the end, my only
concern is how much discomfort is on the road from here to there.

I hope my understanding of the motivations, attitudes, and plans of
those involved are incorrect.  But these understandings are driven by
the actions of the foundation staff, they are positions I'd take with
respect to anyone engaging in the same actions.  They can be most
easily remedied by the staff collaborating with the community as an
equal parter, and not mealy as a source of optional feedback.  Ten
thousand words could not carry the impact of just a few actions.