Hi all,
it seems that the WMF evaluation department has once again put together an
evaluation of Wiki Loves Monuments. Out of curiosity, were any of the
organizers involved in this? A quick glance suggests some factual errors,
and again a big focus on assuming WLM is a consistent project, that is
similar in each country (while in reality it is a diverse collection of
projects, tailored to the needs of each country, by its community) and with
a focus towards number crunching.
Statements that begin with 'the average Wiki Loves Monuments
implementation/contest' make my eyes bleed... Did anyone make a more
thorough analysis of the report?
Best,
Lodewijk
Greetings Wiki Loves Monuments list members,
We really appreciate all your interest in this report. We regularly seek
program leader input to interpreting the data and developing next steps for
learning from case examples. This applies to volunteer program leaders and
grantees and we encourage continued participation in this shared learning
effort. All our channels are open for you to choose how to reach us. The
goal of our team and our reports is to serve movement partners, so we want
to make sure we’re hearing and responding to your main concerns about this
year’s iteration of the Evaluation Reports.
That said, a lot of this thread seemed to be based on misunderstandings. We
wanted to clear some of them up, particularly around the report’s
background and inputs:
(1) Data collection efforts
Several people in this thread have asked how we gathered the data that we
used in these reports. We got data from the voluntary data collection
survey[1], grant reports and their linked event pages, blogs or
supplemental reporting, and by using online tools that are also available
to the community[2]. This year’s project was first announced in September
with a clear outline of the metrics sought. Data collection and input of
metrics was open September through December initially and then extended
through February. The extension to February was to extend direct reporting
inputs two months longer to allow for grantees, whose program data were
first mined from their grant reports, time to connect us to specific
program leaders to fill in the gaps [3], as well as last call to the
community along with our published list of identified programs January
2015.
(2) Data limitations
Some people in the thread have been concerned about the limitations of the
data. We agree that we should be transparent about this, so each report has
a special page that reviews the limitations of the data captured.
Importantly, the Wiki Loves Monuments evaluation report is part of an
expanded folio of the beta reports [4] modeled and discussed last year. As
a set of reports we present overall limitations to the reporting and issues
with data access across each program [5] in the overview of the reporting
[6]. In those sections, we explicitly present the response rates of program
leaders who reported directly, for Wiki Loves Monuments, that portion is
39% who program leaders which report directly.
(3) Diversity of goals
The issue of diverse goals for programs is also included among limitations
overall and highlighted on the Wiki Loves Monuments limitations page [7]
where we point out that, yes, eight different goals were selected by at
least 50% of those reporting directly. These reports are part of a
discovery process with which we have engaged in on-going dialogue about
challenges with metrics for quality, tools accessibility, tracking and
privacy issues, issues with valuation across different socio-economic
contexts, varied interests and foci, and other complexities of measuring
impact across the movement. We will continue to have those conversations as
we look to improve measurement strategies for understanding movement-wide
efforts and impact.
(4) Over-simplification
Some of you were also concerned about over-simplification, and that nuance
is lost when writing simple summary statements such as “The average Wiki
Loves Monuments contest …” “...hurt ... to see.” We wrote these TL;DRs
explicitly in response to feedback on the beta version of these reports
last year. When we proposed the summaries then, we were told that would be
appreciated. Truthfully, these can be really painful statements to have to
write because we know they are, by definition, over-simplifications.
However, we made that compromise in order to make the information
accessible to many different audiences of readers.
Importantly, rolling up metrics across several different points of program
implementation is a difficult task. By definition it sacrifices complexity,
as does developing easy to digest snip-its of information that are
requested by so many who are inundated by information in their inboxes.
So, yes, if you want the details, please skip them and read the more
detailed narrative, or use them to help guide your interest to where you
wish to read more deeply, there is a lot of data to wade through, we have
worked to make it as accessible as possible. We have tried to format in a
linguistically and visually consistent fashion to make these different
reading routes available, but differentiated, for different reader
preferences. Please feedback on how this is working and continue to share
potential solutions as we are always open to improvements.
This email does not answer every question raised on the thread; since we
expect some of these questions will be asked again in the future we have
outlined the most important questions asked, and answered on the report
talk page [8]. Please let us know if we have overlooked any and join us
there so that we can continue the discussion and have the information
documented in a central location in order to use it in future strategy.
In behalf of the Learning and Evaluation team, thank you for your time and
participation in learning together.
Sincerely,
Jaime Anstee
Links:
[1] Data Collection Announcement and Blog Announcement
https://meta.wikimedia.org/wiki/Grants:Evaluation/News/Round_II_Announcementhttp://blog.wikimedia.org/2014/09/24/quantitative-versus-qualitative-more-f…
[2] Tools (Wikimetrics, GLAMorous, CatScan, Quarry)
http://tools.wmflabs.org/
[3] Evaluation Reports (beta)
https://meta.wikimedia.org/wiki/Grants:Evaluation/Evaluation_reports/2013
<https://meta.wikimedia.org/wiki/Grants:Evaluation/Evaluation_reports/2015/I…>
[4] Blog on “Filling in the Gaps”
http://blog.wikimedia.org/2015/01/29/fill-in-the-gaps/
[5] Overall reporting limitations and data access
https://meta.wikimedia.org/wiki/Grants:Evaluation/Evaluation_reports/2015/L…https://meta.wikimedia.org/wikiEvaluation/Evaluation_reports/2015/Data_acce…
<https://meta.wikimedia.org/wiki/Grants:Evaluation/Evaluation_reports/2015/L…>
[6] Reporting Overview (If you are new to the reports and evaluation
initiative we suggest starting at the Important Definitions page and
working your way right through the other tabs to answer your curiosities:
https://meta.wikimedia.org/wiki/Grants:Evaluation/Evaluation_reports/2015/I…
[7] Wiki Loves Monuments report limitations
https://meta.wikimedia.org/wiki/Grants:Evaluation/Evaluation_reports/2015/W…
[8] Wiki Loves Monuments evaluation report talk page
https://meta.wikimedia.org/wiki/Grants_talk:Evaluation/Evaluation_reports/2…
________________________________________
Jaime Anstee, Ph.D
Program Evaluation Specialist
Wikimedia Foundation
Imagine a world in which every single human being can freely share in the
sum of all knowledge. Help us make it a reality!
https://donate.wikimedia.org
TL;DR: The Learning and Evaluation team at Wikimedia Foundation has
released the first Programs Reports for Wiki Loves Monuments [1] and Other
Photo Events [2]. Learn more about how these programs deliver against the
set goals and comment on what next steps should be! Join the event on Wed
May 6 *[3]*
Hi all,
The Learning and Evaluation team at Wikimedia Foundation is happy to launch
the first two Wikimedia programs reports 2015: Wiki Loves Monuments [1] and
Other Photo Events [2]. As many of you know, the reports are the result of
a collaborative effort that brings together program leaders across the
movement and the L&E team, to better understand the impact Wikimedia
programs have online and offline.
Highlights of the first reports include that media uploaded to Wikimedia
Commons as part of the Wiki Loves Monuments and other photo events (from
September 2013 to September 2014) represent 14% of the media uploaded to
Commons and media uploaded are used in articles at five times the rate of
other media uploaded during the same time period. Read the reports to
understand how many new users are introduced to wiki projects through these
programs, to learn about user retention, and to see how effective photo
events are at expanding and improving content on Wikimedia projects. Unlike
the beta reports, these new deliveries includes a new section: «How this
info can apply to program planning», in the section «Key Findings», to help
program leaders make the most of these reports.
In addition to these two reports, we've published an overview report on the
state of program evaluation and reporting. Specifically, the section
Evaluation Initiative [4] will give readers a snapshot on the state of
learning and evaluation in the movement.
We have come a long way, and we want to thank everyone who has been
involved in different parts of this initiative and especially those who
shared data with us! We still have a lot of work to approach together. We
need program leaders ideas and views to explore possible next steps. Some
potential areas for further investigation of Other Photo Events and Wiki
Loves Monuments are:
-
Do different types of photo events attract different types of users?
-
How can we apply the successes of low-cost, low-scale events to other
contexts?
Join the conversation on the Talk Page of both reports. Additionally, we
will host an event to present the reports and discuss them on *Wednesday,
May 6, at 1500 UTC.* We have heard back from some program leaders that will
not be able to make it, but as many of you know, it is hard to find a time
that works across the movement. We are unable to move this virtual event,
but* it will be recorded and available for the public and commenting
afterwards. *Join the event here:
https://plus.google.com/events/cvc3orpmngdoac8019dvei1tcu8
Watch for the next Wikimedia Programs Reports, including:
-
On-wiki Writing Contest
-
GLAM Content donations
-
Editathons
-
Editing Workshops
-
Wikipedia Education Program
-
Conferences
-
Hackathons
-
Wikipedians in Residence
We look forward to your questions, comments and suggestions. See you on the
Talk Page!
*María Cruz * \\ Community Coordinator, PE&D Team \\ Wikimedia Foundation,
Inc.
mcruz(a)wikimedia.org | : @marianarra_ <https://twitter.com/marianarra_>
*[1] Wiki Loves Monuments Evaluation Report:*
https://meta.wikimedia.org/wiki/Grants:Evaluation/Evaluation_reports/2015/W…
*[2] Other Photo Events Evaluation Report:*
https://meta.wikimedia.org/wiki/Grants:Evaluation/Evaluation_reports/2015/O…
*[3] Virtual event about the reports:*
https://plus.google.com/events/cvc3orpmngdoac8019dvei1tcu8
*[4] Evaluation Initiative:*
https://meta.wikimedia.org/wiki/Grants:Evaluation/Evaluation_reports/2015/E…
*[5] Program Reports (beta):*
https://meta.wikimedia.org/wiki/Grants:Evaluation/Evaluation_reports/2013