Hello Analytics,
It seems to easily happen that a well meant metrics is proposed, but that this is actually not sure to be correlated with something value or (at least with my skills) super hard to measure technically [1].
So I wanted to create some easy-to-graspmaterials, maybe to put them on a poster or in a video… whatever.
Sadly I found a lot of warnings about "vanity metrics" online (recommending $ made instead) but no framework, research or case studies on organizational impact and understanding of metrics and how colleagues can be helped to be empowered to do a basic evaluation of metrics.
*So you have any framework, research or case studies that could help us to teach and empower to evaluate** metrics and their use?*
Here is what I found useful so far:
- https://en.wikipedia.org/wiki/DIKW_pyramid and similar frameworks like https://randomblatherdotcom.files.wordpress.com/2014/04/bmust7rcuaa6ueo-jpg-...
- For the need of organizational sensemaking: Weick, Sensemaking in Organizations
- And, not directly related but providing some inspiration: Robust methods and visualizations by Tuckey, decision making under uncertainty by Gingerenzer. Jan
[1] In which case I prefer no metric at all, and investing into qualitative research instead, till the problem space is better understood, but if metrics are already there it often stays…
Hi Jan,
A few years ago, the Grants (now Community Engagement) dept. at WMF put together a set of metrics https://meta.wikimedia.org/wiki/Grants:Learning_patterns/Calculating_global_metrics designed to measure the impact of different kinds of grant-funded and/or "programmatic" events and activities within the movement. Many of these metrics were built into the Wikimetrics tool https://meta.wikimedia.org/wiki/Grants:Evaluation/Wikimetrics, developed by the Analytics team to help individuals and groups measure and report the impact of their activities themselves.
The metrics we developed aren't perfect, but they were based on research and intended to be accessible and provide coverage for a wide variety of activities related to the Wikimedia movement. So they represent a framework in that respect. To learn more about them, you might ask members of the Learning & Evaluation team https://wikimediafoundation.org/wiki/Staff_and_contractors#Learning_.26_Evaluation .
- J
On Sat, Feb 4, 2017 at 7:34 AM, Jan Dittrich jan.dittrich@wikimedia.de wrote:
Hello Analytics,
It seems to easily happen that a well meant metrics is proposed, but that this is actually not sure to be correlated with something value or (at least with my skills) super hard to measure technically [1].
So I wanted to create some easy-to-graspmaterials, maybe to put them on a poster or in a video… whatever.
Sadly I found a lot of warnings about "vanity metrics" online (recommending $ made instead) but no framework, research or case studies on organizational impact and understanding of metrics and how colleagues can be helped to be empowered to do a basic evaluation of metrics.
*So you have any framework, research or case studies that could help us to teach and empower to evaluate** metrics and their use?*
Here is what I found useful so far:
- https://en.wikipedia.org/wiki/DIKW_pyramid and similar frameworks like
https://randomblatherdotcom.files.wordpress.com/2014/04/ bmust7rcuaa6ueo-jpg-large.jpeg
- For the need of organizational sensemaking: Weick, Sensemaking in
Organizations
- And, not directly related but providing some inspiration: Robust methods
and visualizations by Tuckey, decision making under uncertainty by Gingerenzer. Jan
[1] In which case I prefer no metric at all, and investing into qualitative research instead, till the problem space is better understood, but if metrics are already there it often stays… -- Jan Dittrich UX Design/ User Research
Wikimedia Deutschland e.V. | Tempelhofer Ufer 23-24 | 10963 Berlin Phone: +49 (0)30 219 158 26-0 http://wikimedia.de
Imagine a world, in which every single human being can freely share in the sum of all knowledge. That‘s our commitment.
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Furthering that initial metrics work, Aaron did some more research and came up with a set of standard metrics: https://meta.wikimedia.org/wiki/Research:Standard_metrics
On Sun, Feb 5, 2017 at 3:35 PM, Jonathan Morgan jmorgan@wikimedia.org wrote:
Hi Jan,
A few years ago, the Grants (now Community Engagement) dept. at WMF put together a set of metrics https://meta.wikimedia.org/wiki/Grants:Learning_patterns/Calculating_global_metrics designed to measure the impact of different kinds of grant-funded and/or "programmatic" events and activities within the movement. Many of these metrics were built into the Wikimetrics tool https://meta.wikimedia.org/wiki/Grants:Evaluation/Wikimetrics, developed by the Analytics team to help individuals and groups measure and report the impact of their activities themselves.
The metrics we developed aren't perfect, but they were based on research and intended to be accessible and provide coverage for a wide variety of activities related to the Wikimedia movement. So they represent a framework in that respect. To learn more about them, you might ask members of the Learning & Evaluation team https://wikimediafoundation.org/wiki/Staff_and_contractors#Learning_.26_Evaluation .
- J
On Sat, Feb 4, 2017 at 7:34 AM, Jan Dittrich jan.dittrich@wikimedia.de wrote:
Hello Analytics,
It seems to easily happen that a well meant metrics is proposed, but that this is actually not sure to be correlated with something value or (at least with my skills) super hard to measure technically [1].
So I wanted to create some easy-to-graspmaterials, maybe to put them on a poster or in a video… whatever.
Sadly I found a lot of warnings about "vanity metrics" online (recommending $ made instead) but no framework, research or case studies on organizational impact and understanding of metrics and how colleagues can be helped to be empowered to do a basic evaluation of metrics.
*So you have any framework, research or case studies that could help us to teach and empower to evaluate** metrics and their use?*
Here is what I found useful so far:
- https://en.wikipedia.org/wiki/DIKW_pyramid and similar frameworks like
https://randomblatherdotcom.files.wordpress.com/2014/04/bmus t7rcuaa6ueo-jpg-large.jpeg
- For the need of organizational sensemaking: Weick, Sensemaking in
Organizations
- And, not directly related but providing some inspiration: Robust
methods and visualizations by Tuckey, decision making under uncertainty by Gingerenzer. Jan
[1] In which case I prefer no metric at all, and investing into qualitative research instead, till the problem space is better understood, but if metrics are already there it often stays… -- Jan Dittrich UX Design/ User Research
Wikimedia Deutschland e.V. | Tempelhofer Ufer 23-24 | 10963 Berlin Phone: +49 (0)30 219 158 26-0 http://wikimedia.de
Imagine a world, in which every single human being can freely share in the sum of all knowledge. That‘s our commitment.
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- Jonathan T. Morgan Senior Design Researcher Wikimedia Foundation User:Jmorgan (WMF) https://meta.wikimedia.org/wiki/User:Jmorgan_(WMF)
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics