[WikiEN-l] "Fatally Flawed" -- Internal Britannica Review Tackles Nature Methods

[Steve Bennett]

On 3/22/06, SJ <2.718281828 at gmail.com> wrote:
> Those reports were wrong, however, because Nature's research was invalid. As
> our editors and scholarly advisers have discovered by reviewing the research
> in depth, almost everything about the Nature's investigation was wrong and
> misleading. Dozens of inaccuracies attributed to the Britannica were not
> inaccuracies at all, and a number of the articles Nature examined were not
> even in the Encyclopædia Britannica. The study was so poorly carried out and
> its findings so error-laden that it was completely without merit.

I'm surprised how vague this is. Later he complains that the source
data is not available. But here he just refers to "dozens of
inaccuracies"...hmm.

> *       Rearranging, reediting, and excerpting Britannica articles. Several
> of the "articles" Nature sent its outside reviewers were only sections of,
> or excerpts from Britannica entries. Some were cut and pasted together from
> more than one Britannica article. As a result, Britannica's coverage of
> certain subjects was represented in the study by texts that our editors
> never created, approved or even saw.

That seems like a lot of hot air over a small issue. Claiming that
their editors "never created, approved or even saw" the particular
compilation is really piling on strong words to make an impact.
Whereas what they're actually denying is much weaker. Kind of like "I
did *not* have sexual relations with that woman..."

> *       Mistakenly identifying inaccuracies. The journal claimed to have
> found dozens of inaccuracies in Britannica that didn't exist.

Well, I've had a look at some of their claimed inaccurate inaccuracies
and I'm not that sure. It's very much "No it's not / Yes it is".

> *       Reviewing the wrong texts. They reviewed a number of texts that were
> not even in the encyclopedia.

Do they get more specific than this? "A number" probably turns out to
be 3. Or existed in some previous version of the encyclopaedia that
was outdated 4 days before the survey. I'm suspicious.

> *       Failing to check facts. Nature falsely attributed inaccuracies to
> Britannica based on statements from its reviewers that were themselves
> inaccurate and which Nature's editors failed to verify.

Well, details, please.

> *       Misrepresenting its findings. Even according to Nature's own
> figures, (which grossly exaggerated the number of inaccuracies in
> Britannica) Wikipedia had a third more inaccuracies than Britannica. Yet the
> headline of the journal's report concealed this fact and implied something
> very different.

I vaguely recall they had some sort of weird logarithmic scale, so I
would tend to agree with this one. But actually...wasn't the final
number something like "4 errors per article compared to 3 errors per
article"? Whether 33% more is a lot or not is purely subjective. I
would definitely say it's the same ballpark.

> Britannica also made repeated attempts to obtain from Nature the original
> data on which the study's conclusions were based. We invited Nature's
> editors and management to meet with us to discuss our analysis, but they
> declined.

If true, that's very poor on Nature's part.

> Whatever may have prompted Nature to do such careless and sloppy research,
> it's now time for them to uphold their commitment to good science and
> retract the study immediately. We have urged them strongly to do so.

Uh, a commitment to good science would be publishing the data and
their method so that people can attempt to reproduce it. I tend to
agree that Nature's method was optimistic. Surely more than one
reviewer for each article shoul have been appointed...even better
would have been an open forum where many experts could pick through
each article and argue amongst themselves.

> We have prepared a detailed report that describes Britannica's thorough
> (7,000 words) analysis of the Nature study. I invite you to download it from
> our Web site at www.eb.com.

A lot of it seems to be "We reject this criticism. We have asked our
adviser, and he says we're right." Not exactly convincing stuff.
Particularly when they complain that the original study failed to cite
sources - and with few exceptions, the EB rebuttal doesn't either.

Example: "Britannica response: We do not accept these criticisms
(which are really just one criticism, not two). We have published a
revision of this article that retains the emphasis on supersaturation
rather than the transitional stage of saturation."

By this standard, Wikipedia should reply to nature and say "Thanks for
the criticisms. Our community rejects them. We stand by our original
version". However, we didn't - as I understand, we actually took the
criticisms on board and worked with them. Which kind of demonstrates
the real strength in Wikipedia. Instead of simply "not accepting"
every criticism (as EB does 22 times in their response) to protect our
good name, we, without ego, simply make it better.

I also sort of feel that as an attempted model of good science or
whatever, EB should not be attacking the entire Nature study as
totally "without merit".

Sorry I can't join the IRC chat.

Steve