[WikiEN-l] Reliable sources— some of these babi es are ugly

[Abd ul-Rahman Lomax]

At 04:08 AM 5/23/2010, Ray Saintonge wrote:
>Philip Sandifer wrote:
> > On May 15, 2010, at 10:12 AM, Gregory Maxwell wrote:
> >> [...]I can't say that these points really apply in many cases that we
> >> appear to be applying them: We would reject as reliable sources many
> >> hobbyist blogs (or even webcomics) with a stronger reputation to
> >> preserve, less obviously-compromised motivations, and _significantly_
> >> greater circulation than some obscure corner of Fox News's online
> >> product.  What can be the explanation for this discrepancy?
> >>
> > Two reasons. 1) Egregious anti-expert bias. 2) A fundamental 
> misunderstanding of the nature of the written record of humanity.
> >
> > 1) Our policies are explicitly and deliberately written to try to 
> allow content decisions to be made without any actual knowledge of 
> the subject. That is, we have actively tried to write policy that 
> rejects any thinking about sources beyond the surface level 
> readings, and that take as a premise that, given a large enough 
> pile of books, anybody can adequately write or edit an article on 
> any topic. This premise is dubious at best.

Indeed. We needed the adhocracy to rapidly develop a broad project, 
but we failed to incorporate and implement procedures to move beyond 
that, imagining, I suspect, that we could do it "later." But by the 
time "later" arrived, constituencies had formed that were broad 
enough, at the core, to prevent the necessary extensions. It's a 
common problem with organizations that work when small but break down 
as the scale increases.

Given that projects based on the idea that experts -- i.e., 
professionals -- should control articles, for free distribution, 
failed, what would have remained was "amateur experts," who will, 
indeed, freely contribute content. But an impolite term for "amateur 
expert" is "POV-pusher." (The parallel term for professional expert 
is "COI POV-pusher." Experts are rarely neutral!) Experts, however, 
will not make stupid misinterpretations of sources, as routinely 
happens with non-experts, and as even often happens with professional 
media, as anyone who is closely familiar with a topic has become 
accustomed to noticing when an article appears in a newspaper on what 
they know.

The potential is for Wikipedia to do better than professional media, 
but it would require structure that was not developed. The key is 
that experts, indeed, should not control articles, but those who will 
use the articles should. An expert opinion that is not effective in 
communication to non-experts is useless except perhaps as a salvo in 
a battle between experts. Those of us who *need* a neutral 
encyclopedia will want to insist that expert opinin is used to 
prevent stupid mistakes, but that where there is variation in expert 
opinion, it is fairly represented so that we can understand the range 
of views. Wikipedia sometimes does well with this, but not when 
political debates and entrenched positions exist between experts.

I argued, in RfAr/Abd-William M. Connolley, that those claiming 
expertise should be considered COI, which was preposterously taken as 
asserting that they should be sanctioned, a revealing assumption 
indeed. No, they should be protected, simply not allowed to *control* 
articles, but rather invited and encouraged to *advise* the 
community, and to point out supporting sources where possible. It 
will not always be possible, or it may be possible, but only with 
more work than we can expect an expert to contribute. At one point, 
there was some practice of, where a controversy broke out in an 
article where expert opinion could be valuable, seeking out expert 
opinion by emailing experts. That's what a professional encyclopedia 
would have done. In peer-reviewed journals, "private communication 
with X" as a source is often seen. We did use and should have 
continued to use this, as equivalent to unreviewed opinion from 
someone reasonably expected to have an informed opinion. (It's the 
same as a blog opinion by a recognized expert, an example of where 
self-published work is sometimes allowed.)

None of this will work without true consensus process in place. The 
famous "anti-expert bias" of Wikipedia was a result of failing to 
develop such process. Obviously, genuine consensus process should not 
and would not exclude experts!

>I don't believe that there is such a thing as a reliable source. Most
>people will believe exactly what they want to believe, with a remarkable
>preference for not having their beliefs encumbered by facts.

That's the default, without deliberative process. Because 
deliberative process can be tedious and can more or less force people 
to re-examine their beliefs (or abandon the discussion and decision, 
or turn it into a battle and personality conflict), and because such 
process was not formalized early on, it has been and continues to be 
interrupted, and those with special privileges, or with bias that 
they do not wish to examine, often see it as a waste of time.

Ray is correct in the sense that no source is fully reliable, just as 
no editor is fully neutral, unless ignorant, which sets up quite a 
conflict in an adhocracy that assumes neutrality for adequate function!

Relative reliability and relative neutrality are possible, though. In 
any case, for Wikipedia, "reliable" is a term of art and does not 
mean that a fact or unattributed opinion found in such a source 
(deemed "reliable" because of the nature of the publication) is 
necessarily true, just that it is, ipso facto, sufficiently notable 
that neutrality requires it be included in the mix of a neutral 
article, directly or indirectly, not excluded unless possibly 
redundant to what is included. That's policy, in fact, confirmed by 
ArbComm, but when we stray into the obsessions of the anti-fringe 
interest groups, it has been commonly ignored.

Neutrality is fundamental Wikipedia policy, but an error was made, an 
assumption that any editor or even group of editors, unless very 
broadly inclusive, can solely judge "neutrality." Bias in text may 
only be detectable by those who have a contrary bias or POV. Thus the 
fundamental policy required consensus process, not merely majority or 
even supermajority rule. "Consensus process" does not mean that 
everyone, in the end, agrees, but it requires that such agreement be 
earnestly sought, because whenever a view is excluded and 
disregarded, there is a risk to neutrality. If neutrality is 
important, and it certainly is, then, the goal of full consensus must 
be respected. And I certainly found, in my on-wiki experience, that 
it is not, and in RfAr/Abd-William M. Connolley, I was astonished to 
see so many administrators openly express that it was preposterous to 
seek full consensus. I did not think they would be so bold.