Hi all,
This was posed as a question on the GLAM Wiki Global Telegram group, but I thought I'd send it to the WREN list for thoughts and comments. Feel free to respond in email or at the original thread: https://t.me/c/1723589369/4942
This was in response to the current draft of the annual plan, and a discussion around Selena Deckelmann's (WMF CPTO) response to resourcing Wikimedia Commons.
Thanks. -Andrew
----
I'd like to put out a question that may be key to understanding the debate seen around "knowledge dissemination," as was mentioned in Selena Deckelmann's response to the resourcing Commons statement: https://commons.wikimedia.org/wiki/Commons_talk:Media_knowledge_beyond_Wikip...
How do you evaluate or measure the impact of open content?
By its very nature, open content (text, data, code, media) is out of the control of its originator. The editor, uploader, file host makes it available, and others copy it and use it somewhere down the path of replication. Therefore, evaluating the use and impact of any open content can be extremely hard – free/libre content encourages dissemination, but removes the ability to measure it. It may be possible through secondary indicators (e.g. Github stars) or best-effort research (e.g. Google search for image credits, hopefully finding mentions of Commons or a user). However, all too often measuring impact is a qualitative educated guess. This is not a new problem. But I believe this lack of sound methodology for evaluating impact, and our lack of evolving our thinking, has become a blocker in multiple ways.
Some questions to ponder: 1) Some GLAM partners have mentioned this problem specifically, and it would be interesting to hear if others heard the same issues. For example, Cleveland Museum of Art were early adopters of open access, uploading Commons images and creating Wikidata items, but are finding it hard to justify more activity without understanding the impact of these efforts. How might we do better to discover the impact of these GLAM engagements outside of our limited Commons/Wikipedia metrics?
2) We know that both Commons and Wikidata have superlatives of their own – Wiki Loves Monuments is the world's largest public photo contest and Wikidata has been the project that racks up the most edits of all the Wikimedia projects. But outside of our Wikimedia servers, it's hard to tell. We know anecdotally that libraries and archives use Wikidata extensively now for linked open data, with organizations like LD4 and IFLA asking for more Wikidata engagement. All major library databases in the world use Wikidata Q identifiers. But how might we capture that impact better, and on a regular basis?
3) Finally, we come to the essay and Talk page mentioned above. My concern is that since Wikipedia metrics are so easily available and are of such huge magnitude, the appeal to be fixated on them is overwhelming. It's easy to put Commons and Wikidata in a lesser corner as we don't have good methods to estimate their downstream impact. Much of the value of Commons and Wikidata are not people visiting the Commons image page, or the Wikidata item page, but rather the linkages and re-use in the open ecosystem – federated search, authority control records, multimedia visualizations. And because that's hard to measure, we risk always putting Commons and Wikidata into second-class status, when they should be seen as a different beast.
Apologies for the long-winded examples, but it does come down to this:
How might we make a more complete evaluation of the impact of Wikimedia's open content, so we can make better decisions for the future?