[Foundation-l] Five-year WMF targets exclude non-Wikipedia projects

[Sue Gardner]

Hi folks,

Sorry to have been absent from this discussion thus far: I didn't
realize Sam was going to post the targets when he did, and so I am
playing a little catch-up here.

Below are some questions and answers re the targets that might be
helpful for the discussion. (Erik wrote most of this, and I've just
now added a few bits.)  If you read this and there are still issues
that you want addressed, please just say so :-)

Thanks,
Sue



http://wikimediafoundation.org/wiki/Resolution:Five-year_targets

What’s the purpose of these five-year targets?

The July 2015 targets approved by the Board represent [[big hairy
audacious goal]]s for Wikimedia. They are intended to reflect
Wikimedia’s core mission -- “to empower and engage people around the
world to collect and develop free educational content and to
disseminate it effectively and globally” -- and to capture it in terms
of concrete, ambitious outcomes that are very hard --but ideally not
impossible-- to achieve.

The development of very ambitious five-year targets is intended to
help inspire and focus energy across the Wikimedia movement, in
connection with the priorities identified through Wikimedia’s
strategic planning process. They’re also intended to help persuade
people who aren’t yet active members of the Wikimedia movement (for
example, readers who don’t yet edit or donate, grant-making
institutions, “GLAM” organizations) to get involved and support us.

How do you expect these targets to be used?

These targets will be part of the printed “summary” version of the
strategy plan that we will distribute internally and to external
interested parties -- e.g., Wikipedia Academy attendees, grant-making
institutions, etc. We will give out the document, including the
targets, so that people understand where the Wikimedia movement is
focusing its energy, and how they can help.

The targets will also help us, we hope, to focus our own work, and
measure whether we’re being successful. In the context of an overall
Wikimedia dashboard, we can begin to highlight if key performance
indicators are significantly deviating from our expectations -- e.g.
if overall article growth flattens, or the number of editors declines.

Are these the only targets used by the Wikimedia Foundation?

No. These five targets are called out as very ambitious long-term
outcomes which, if we achieve them together, will indicate that
Wikimedia has made great strides in serving its mission. They are
intentionally high level, focused on information and the people who
develop and receive it, as opposed to operations.

There are other key performance indicators which we must examine on an
ongoing basis, including but not limited to:
 * engagement and retention of the editor community
 * site uptime and load times in different geographies
 * financial health of the Wikimedia movement
 * availability of secure off-site copies of all data
 * number and quality of multimedia files in all our projects
 * number and quality of information in Wikimedia’s other projects
 * demographic composition of the editor community
 * our collective ability to develop and operationalize innovative technology

The Wikimedia report-card, at http://stats.wikimedia.org/reportcard/ ,
is our primary instrument for tracking key performance indicators; its
formats and the indicators which are included are still evolving. The
Wikimedia report-card is much more detailed than the five-year
targets: it tracks more information at a more granular level, more
frequently.

How were these targets developed?

The development of five-year targets was the last major piece of work
that needed to be concluded before the strategy project wrapped up.

To that end, the targets were developed primarily by the staff of the
Wikimedia Foundation (mainly Sue Gardner, Erik Moeller, and Barry
Newstead), for Board approval. The first source for material was the
discussions held on the strategy wiki, focused on performance
indicators and goals (e.g.,
<http://strategy.wikimedia.org/wiki/Strategic_Plan/Movement_Priorities>).
Another source was the set of internal metrics that the Wikimedia
community has used for a long time, including “number of articles” and
“active editors”, as well as first baseline estimates for gender
participation and country-of-origin breakdown that have been
established more recently through surveys, log analysis, and other
methods.

People’s general views about target-setting, as well as their
assessment of the value of different possible measures, were surfaced
through surveys of foundation-l and internal-l readers, Advisory Board
members, Board members and staff members [1]. That helped shape both
our general approach to target-setting, and the actual measures and
numbers we wanted to use. Once a draft set of targets was created, Sue
gave it to the Board, where it was discussed at some length and then
approved.

[1] See Sue’s blog post at
<http://suegardner.org/2010/08/16/how-wikimedia-will-measure-success-over-the-next-five-years/>
for some background.

Can these targets be modified?

While other key performance indicators (see above) are intended to be
flexibly adjusted as we better understand which indicators are useful
and which ones aren’t, the five-year targets voted on by the Board are
intended to be stable. That’s why they are so high-level. The
underlying measurements and methodologies for each target will
continue to be refined over time -- more so in categories where we
lack clear and consistent baselines, such as article quality.

Why was “number of Wikipedia articles” used as a target for “amount of
information we offer”?

We were looking for a single, clearly understandable indicator that we
weren’t just reaching more people, growing our community and
increasing its diversity, and improving perceived quality of content,
but that we were also successfully expanding the breadth and depth of
information available to readers.

We fully understand that the number of Wikipedia articles is an
imperfect measure -- it’s especially imperfect if taken by itself. On
the other hand, it’s fairly well understood what a Wikipedia article
is, what it isn’t, and what the potential challenges with counting
Wikipedia articles are (such as mass-creation of articles based on
some data source): we’ve used article counts in Wikipedia since the
project started, and have developed a collective expertise how to
manage and how to interpret them.

Taken by itself, the number of Wikipedia articles answers one
question, and one question only: How likely is it that Wikimedia’s
flagship project, Wikipedia, is going to have any information on a
given term? It doesn’t answer whether the information is useful, of
high quality, or even whether it should be in Wikipedia at all. But
the other targets help to answer that question: We are measuring
whether we are increasing the reach of our projects, the perceived
quality of the information provided, and the number and diversity of
contributors. If we succeed along all those dimensions, the number of
Wikipedia articles is a useful indicator of our overall breadth and
depth. This is not to diminish the work done in other categories of
content (such as media files), or the work done in Wikipedia’s sister
projects. The “number of Wikipedia articles” is expected to be useful
as an _indicator_ of the overall amount of information we offer: that
is its primary usefulness.

What does a “25% increase in quality” actually mean?

We’ve not yet established which assessment systems will be the most
useful, scalable, and effective, and until we have done so, this
measure should be seen principally as a placeholder for a large,
ambitious, relative increase in perceived quality of a large number of
assessed articles over the five-year time period (most simply, in the
context of a scoring system like
<http://www.mediawiki.org/wiki/Article_feedback/Public_Policy_Pilot>,
a 25% increase of average scores for assessed articles over the
five-year time period).

Wikimedia projects to-date have primarily used internal measures of
quality (“good articles”, “featured articles”, patrolled edits,
WikiProject classification, etc.). These measures are of critical
importance to the organization of Wikimedia’s editorial efforts. At
the same time, they only measure what we, ourselves, think about the
quality of the product that we’ve built. It’s equally important that
we obtain measures of what others think about the quality provided --
this includes our audience, but it also includes individuals with
verifiable expertise regarding the subject matter domain(s) an article
relates to.



-- 
Sue Gardner
Executive Director
Wikimedia Foundation

415 839 6885 office
415 816 9967 cell

Imagine a world in which every single human being can freely share in
the sum of all knowledge.  Help us make it a reality!

http://wikimediafoundation.org/wiki/Donate