WikiLovesMonuments May 2015

wikilovesmonuments@lists.wikimedia.org

38 participants
13 discussions

'evaluation'
by Lodewijk 07 May '15

07 May '15

Hi all, it seems that the WMF evaluation department has once again put together an evaluation of Wiki Loves Monuments. Out of curiosity, were any of the organizers involved in this? A quick glance suggests some factual errors, and again a big focus on assuming WLM is a consistent project, that is similar in each country (while in reality it is a diverse collection of projects, tailored to the needs of each country, by its community) and with a focus towards number crunching. Statements that begin with 'the average Wiki Loves Monuments implementation/contest' make my eyes bleed... Did anyone make a more thorough analysis of the report? Best, Lodewijk

16 30

'evaluation'
by Jaime Anstee 04 May '15

04 May '15

Greetings Wiki Loves Monuments list members, We really appreciate all your interest in this report. We regularly seek program leader input to interpreting the data and developing next steps for learning from case examples. This applies to volunteer program leaders and grantees and we encourage continued participation in this shared learning effort. All our channels are open for you to choose how to reach us. The goal of our team and our reports is to serve movement partners, so we want to make sure we’re hearing and responding to your main concerns about this year’s iteration of the Evaluation Reports. That said, a lot of this thread seemed to be based on misunderstandings. We wanted to clear some of them up, particularly around the report’s background and inputs: (1) Data collection efforts Several people in this thread have asked how we gathered the data that we used in these reports. We got data from the voluntary data collection survey[1], grant reports and their linked event pages, blogs or supplemental reporting, and by using online tools that are also available to the community[2]. This year’s project was first announced in September with a clear outline of the metrics sought. Data collection and input of metrics was open September through December initially and then extended through February. The extension to February was to extend direct reporting inputs two months longer to allow for grantees, whose program data were first mined from their grant reports, time to connect us to specific program leaders to fill in the gaps [3], as well as last call to the community along with our published list of identified programs January 2015. (2) Data limitations Some people in the thread have been concerned about the limitations of the data. We agree that we should be transparent about this, so each report has a special page that reviews the limitations of the data captured. Importantly, the Wiki Loves Monuments evaluation report is part of an expanded folio of the beta reports [4] modeled and discussed last year. As a set of reports we present overall limitations to the reporting and issues with data access across each program [5] in the overview of the reporting [6]. In those sections, we explicitly present the response rates of program leaders who reported directly, for Wiki Loves Monuments, that portion is 39% who program leaders which report directly. (3) Diversity of goals The issue of diverse goals for programs is also included among limitations overall and highlighted on the Wiki Loves Monuments limitations page [7] where we point out that, yes, eight different goals were selected by at least 50% of those reporting directly. These reports are part of a discovery process with which we have engaged in on-going dialogue about challenges with metrics for quality, tools accessibility, tracking and privacy issues, issues with valuation across different socio-economic contexts, varied interests and foci, and other complexities of measuring impact across the movement. We will continue to have those conversations as we look to improve measurement strategies for understanding movement-wide efforts and impact. (4) Over-simplification Some of you were also concerned about over-simplification, and that nuance is lost when writing simple summary statements such as “The average Wiki Loves Monuments contest …” “...hurt ... to see.” We wrote these TL;DRs explicitly in response to feedback on the beta version of these reports last year. When we proposed the summaries then, we were told that would be appreciated. Truthfully, these can be really painful statements to have to write because we know they are, by definition, over-simplifications. However, we made that compromise in order to make the information accessible to many different audiences of readers. Importantly, rolling up metrics across several different points of program implementation is a difficult task. By definition it sacrifices complexity, as does developing easy to digest snip-its of information that are requested by so many who are inundated by information in their inboxes. So, yes, if you want the details, please skip them and read the more detailed narrative, or use them to help guide your interest to where you wish to read more deeply, there is a lot of data to wade through, we have worked to make it as accessible as possible. We have tried to format in a linguistically and visually consistent fashion to make these different reading routes available, but differentiated, for different reader preferences. Please feedback on how this is working and continue to share potential solutions as we are always open to improvements. This email does not answer every question raised on the thread; since we expect some of these questions will be asked again in the future we have outlined the most important questions asked, and answered on the report talk page [8]. Please let us know if we have overlooked any and join us there so that we can continue the discussion and have the information documented in a central location in order to use it in future strategy. In behalf of the Learning and Evaluation team, thank you for your time and participation in learning together. Sincerely, Jaime Anstee Links: [1] Data Collection Announcement and Blog Announcement https://meta.wikimedia.org/wiki/Grants:Evaluation/News/Round_II_Announcement http://blog.wikimedia.org/2014/09/24/quantitative-versus-qualitative-more-f… [2] Tools (Wikimetrics, GLAMorous, CatScan, Quarry) http://tools.wmflabs.org/ [3] Evaluation Reports (beta) https://meta.wikimedia.org/wiki/Grants:Evaluation/Evaluation_reports/2013 <https://meta.wikimedia.org/wiki/Grants:Evaluation/Evaluation_reports/2015/I…> [4] Blog on “Filling in the Gaps” http://blog.wikimedia.org/2015/01/29/fill-in-the-gaps/ [5] Overall reporting limitations and data access https://meta.wikimedia.org/wiki/Grants:Evaluation/Evaluation_reports/2015/L… https://meta.wikimedia.org/wikiEvaluation/Evaluation_reports/2015/Data_acce… <https://meta.wikimedia.org/wiki/Grants:Evaluation/Evaluation_reports/2015/L…> [6] Reporting Overview (If you are new to the reports and evaluation initiative we suggest starting at the Important Definitions page and working your way right through the other tabs to answer your curiosities: https://meta.wikimedia.org/wiki/Grants:Evaluation/Evaluation_reports/2015/I… [7] Wiki Loves Monuments report limitations https://meta.wikimedia.org/wiki/Grants:Evaluation/Evaluation_reports/2015/W… [8] Wiki Loves Monuments evaluation report talk page https://meta.wikimedia.org/wiki/Grants_talk:Evaluation/Evaluation_reports/2… ________________________________________ Jaime Anstee, Ph.D Program Evaluation Specialist Wikimedia Foundation Imagine a world in which every single human being can freely share in the sum of all knowledge. Help us make it a reality! https://donate.wikimedia.org

1 0

WLM and Other photo events Evaluation Reports 2015: join the conversation!
by Maria Cruz 03 May '15

03 May '15

TL;DR: The Learning and Evaluation team at Wikimedia Foundation has released the first Programs Reports for Wiki Loves Monuments [1] and Other Photo Events [2]. Learn more about how these programs deliver against the set goals and comment on what next steps should be! Join the event on Wed May 6 *[3]* Hi all, The Learning and Evaluation team at Wikimedia Foundation is happy to launch the first two Wikimedia programs reports 2015: Wiki Loves Monuments [1] and Other Photo Events [2]. As many of you know, the reports are the result of a collaborative effort that brings together program leaders across the movement and the L&E team, to better understand the impact Wikimedia programs have online and offline. Highlights of the first reports include that media uploaded to Wikimedia Commons as part of the Wiki Loves Monuments and other photo events (from September 2013 to September 2014) represent 14% of the media uploaded to Commons and media uploaded are used in articles at five times the rate of other media uploaded during the same time period. Read the reports to understand how many new users are introduced to wiki projects through these programs, to learn about user retention, and to see how effective photo events are at expanding and improving content on Wikimedia projects. Unlike the beta reports, these new deliveries includes a new section: «How this info can apply to program planning», in the section «Key Findings», to help program leaders make the most of these reports. In addition to these two reports, we've published an overview report on the state of program evaluation and reporting. Specifically, the section Evaluation Initiative [4] will give readers a snapshot on the state of learning and evaluation in the movement. We have come a long way, and we want to thank everyone who has been involved in different parts of this initiative and especially those who shared data with us! We still have a lot of work to approach together. We need program leaders ideas and views to explore possible next steps. Some potential areas for further investigation of Other Photo Events and Wiki Loves Monuments are: - Do different types of photo events attract different types of users? - How can we apply the successes of low-cost, low-scale events to other contexts? Join the conversation on the Talk Page of both reports. Additionally, we will host an event to present the reports and discuss them on *Wednesday, May 6, at 1500 UTC.* We have heard back from some program leaders that will not be able to make it, but as many of you know, it is hard to find a time that works across the movement. We are unable to move this virtual event, but* it will be recorded and available for the public and commenting afterwards. *Join the event here: https://plus.google.com/events/cvc3orpmngdoac8019dvei1tcu8 Watch for the next Wikimedia Programs Reports, including: - On-wiki Writing Contest - GLAM Content donations - Editathons - Editing Workshops - Wikipedia Education Program - Conferences - Hackathons - Wikipedians in Residence We look forward to your questions, comments and suggestions. See you on the Talk Page! *María Cruz * \\ Community Coordinator, PE&D Team \\ Wikimedia Foundation, Inc. mcruz(a)wikimedia.org | : @marianarra_ <https://twitter.com/marianarra_> *[1] Wiki Loves Monuments Evaluation Report:* https://meta.wikimedia.org/wiki/Grants:Evaluation/Evaluation_reports/2015/W… *[2] Other Photo Events Evaluation Report:* https://meta.wikimedia.org/wiki/Grants:Evaluation/Evaluation_reports/2015/O… *[3] Virtual event about the reports:* https://plus.google.com/events/cvc3orpmngdoac8019dvei1tcu8 *[4] Evaluation Initiative:* https://meta.wikimedia.org/wiki/Grants:Evaluation/Evaluation_reports/2015/E… *[5] Program Reports (beta):* https://meta.wikimedia.org/wiki/Grants:Evaluation/Evaluation_reports/2013

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

WikiLovesMonuments May 2015