Hello everyone - apologies for cross-posting! *TL;DR*: We would like your feedback on our Metrics Kit project. Please have a look and comment on Meta-Wiki: https://meta.wikimedia.org/wiki/Community_health_initiative/Metrics_kit
The Wikimedia Foundation's Trust and Safety team, in collaboration with the Community Health Initiative, is working on a Metrics Kit designed to measure the relative "health"[1] of various communities that make up the Wikimedia movement: https://meta.wikimedia.org/wiki/Community_health_initiative/Metrics_kit
The ultimate outcome will be a public suite of statistics and data looking at various aspects of Wikimedia project communities. This could be used by both community members to make decisions on their community direction and Wikimedia Foundation staff to point anti-harassment tool development in the right direction.
We have a set of metrics we are thinking about including in the kit, ranging from the ratio of active users to active administrators, administrator confidence levels, and off-wiki factors such as freedom to participate. It's ambitious, and our methods of collecting such data will vary.
Right now, we'd like to know: * Which metrics make sense to collect? Which don't? What are we missing? * Where would such a tool ideally be hosted? Where would you normally look for statistics like these? * We are aware of the overlap in scope between this and Wikistats < https://stats.wikimedia.org/v2/#/all-projects%3E — how might these tools coexist?
Your opinions will help to guide this project going forward. We'll be reaching out at different stages of this project, so if you're interested in direct messaging going forward, please feel free to indicate your interest by signing up on the consultation page.
Looking forward to reading your thoughts.
best, Joe
P.S.: Please feel free to CC me in conversations that might happen on this list!
[1] What do we mean by "health"? There is no standard definition of what makes a Wikimedia community "healthy", but there are many indicators that highlight where a wiki is doing well, and where it could improve. This project aims to provide a variety of useful data points that will inform community decisions that will benefit from objective data.
-- *Joe Sutherland* (he/him or they/them) Trust and Safety Specialist Wikimedia Foundation joesutherland.rocks
Hi Joe,
I think this project is fundamental. I'm glad you are working on it.
I have researched this topic in my PhD thesis and I went through a review of the online communities engagement literature.
Few ideas for metrics: - Contributions inequality measurements (gini coefficients as a start). - Multilingual editors contributions (to see whether they see Wikipedia as a global project or prefer to focus on one language). - Core-periphery social interactions (admins-newbees, in order to detect communities more prone to mentoring) - Rate of newbies completing the first article, rate of newbies completing the first translation, etc. - Recency measures for newbies (different measures on editor retention). - Community/functional roles renewal (admin, autopatrolled, etc. to see how good a community is at renewing its core along the years).
I'd be happy to further discuss the topic. At your disposal. Best regards,
Marc Miquel
El dv., 5 d’oct. 2018 a les 23:29, Joe Sutherland (< jsutherland@wikimedia.org>) va escriure:
Hello everyone - apologies for cross-posting! *TL;DR*: We would like your feedback on our Metrics Kit project. Please have a look and comment on Meta-Wiki: https://meta.wikimedia.org/wiki/Community_health_initiative/Metrics_kit
The Wikimedia Foundation's Trust and Safety team, in collaboration with the Community Health Initiative, is working on a Metrics Kit designed to measure the relative "health"[1] of various communities that make up the Wikimedia movement: https://meta.wikimedia.org/wiki/Community_health_initiative/Metrics_kit
The ultimate outcome will be a public suite of statistics and data looking at various aspects of Wikimedia project communities. This could be used by both community members to make decisions on their community direction and Wikimedia Foundation staff to point anti-harassment tool development in the right direction.
We have a set of metrics we are thinking about including in the kit, ranging from the ratio of active users to active administrators, administrator confidence levels, and off-wiki factors such as freedom to participate. It's ambitious, and our methods of collecting such data will vary.
Right now, we'd like to know:
- Which metrics make sense to collect? Which don't? What are we missing?
- Where would such a tool ideally be hosted? Where would you normally look
for statistics like these?
- We are aware of the overlap in scope between this and Wikistats <
https://stats.wikimedia.org/v2/#/all-projects%3E — how might these tools coexist?
Your opinions will help to guide this project going forward. We'll be reaching out at different stages of this project, so if you're interested in direct messaging going forward, please feel free to indicate your interest by signing up on the consultation page.
Looking forward to reading your thoughts.
best, Joe
P.S.: Please feel free to CC me in conversations that might happen on this list!
[1] What do we mean by "health"? There is no standard definition of what makes a Wikimedia community "healthy", but there are many indicators that highlight where a wiki is doing well, and where it could improve. This project aims to provide a variety of useful data points that will inform community decisions that will benefit from objective data.
-- *Joe Sutherland* (he/him or they/them) Trust and Safety Specialist Wikimedia Foundation joesutherland.rocks _______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Thank you for opening the discussion. In our research group https://grasia.fdi.ucm.es/newmain/language/en/, we are working specifically on this.
You can get a lot of good ideas and pointers to other research from the Wikimedia research page https://research.wikimedia.org/community-health.html and from the last Inspire Campaign of Wikimedia, which it was precisely about Measuring editing community health: https://meta.m.wikimedia.org/wiki/Grants:IdeaLab/Inspire
Thank you also Marc for sharing your ideas, they are very interesting. We have already been working with inequality metrics http://www.opensym.org/wp-content/uploads/2018/07/OpenSym2018_paper_31-1.pdf .
El vie., 19 oct. 2018 a las 16:35, Marc Miquel (marcmiquel@gmail.com) escribió:
Hi Joe,
I think this project is fundamental. I'm glad you are working on it.
I have researched this topic in my PhD thesis and I went through a review of the online communities engagement literature.
Few ideas for metrics:
- Contributions inequality measurements (gini coefficients as a start).
- Multilingual editors contributions (to see whether they see Wikipedia as
a global project or prefer to focus on one language).
- Core-periphery social interactions (admins-newbees, in order to detect
communities more prone to mentoring)
- Rate of newbies completing the first article, rate of newbies completing
the first translation, etc.
- Recency measures for newbies (different measures on editor retention).
- Community/functional roles renewal (admin, autopatrolled, etc. to see
how good a community is at renewing its core along the years).
I'd be happy to further discuss the topic. At your disposal. Best regards,
Marc Miquel
El dv., 5 d’oct. 2018 a les 23:29, Joe Sutherland (< jsutherland@wikimedia.org>) va escriure:
Hello everyone - apologies for cross-posting! *TL;DR*: We would like your feedback on our Metrics Kit project. Please have a look and comment on Meta-Wiki: https://meta.wikimedia.org/wiki/Community_health_initiative/Metrics_kit
The Wikimedia Foundation's Trust and Safety team, in collaboration with the Community Health Initiative, is working on a Metrics Kit designed to measure the relative "health"[1] of various communities that make up the Wikimedia movement: https://meta.wikimedia.org/wiki/Community_health_initiative/Metrics_kit
The ultimate outcome will be a public suite of statistics and data looking at various aspects of Wikimedia project communities. This could be used by both community members to make decisions on their community direction and Wikimedia Foundation staff to point anti-harassment tool development in the right direction.
We have a set of metrics we are thinking about including in the kit, ranging from the ratio of active users to active administrators, administrator confidence levels, and off-wiki factors such as freedom to participate. It's ambitious, and our methods of collecting such data will vary.
Right now, we'd like to know:
- Which metrics make sense to collect? Which don't? What are we missing?
- Where would such a tool ideally be hosted? Where would you normally
look for statistics like these?
- We are aware of the overlap in scope between this and Wikistats <
https://stats.wikimedia.org/v2/#/all-projects%3E — how might these tools coexist?
Your opinions will help to guide this project going forward. We'll be reaching out at different stages of this project, so if you're interested in direct messaging going forward, please feel free to indicate your interest by signing up on the consultation page.
Looking forward to reading your thoughts.
best, Joe
P.S.: Please feel free to CC me in conversations that might happen on this list!
[1] What do we mean by "health"? There is no standard definition of what makes a Wikimedia community "healthy", but there are many indicators that highlight where a wiki is doing well, and where it could improve. This project aims to provide a variety of useful data points that will inform community decisions that will benefit from objective data.
-- *Joe Sutherland* (he/him or they/them) Trust and Safety Specialist Wikimedia Foundation joesutherland.rocks _______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
This seems a start towards way to message "community health" that anyone can grasp: https://meta.m.wikimedia.org/wiki/Grants:IdeaLab/Health_rating_radio_button_...
On Mon, Oct 22, 2018 at 4:10 AM ABEL SERRANO JUSTE abeserra@ucm.es wrote:
Thank you for opening the discussion. In our research group https://grasia.fdi.ucm.es/newmain/language/en/, we are working specifically on this.
You can get a lot of good ideas and pointers to other research from the Wikimedia research page https://research.wikimedia.org/community-health.html and from the last Inspire Campaign of Wikimedia, which it was precisely about Measuring editing community health: https://meta.m.wikimedia.org/wiki/Grants:IdeaLab/Inspire
Thank you also Marc for sharing your ideas, they are very interesting. We have already been working with inequality metrics http://www.opensym.org/wp-content/uploads/2018/07/OpenSym2018_paper_31-1.pdf .
El vie., 19 oct. 2018 a las 16:35, Marc Miquel (marcmiquel@gmail.com) escribió:
Hi Joe,
I think this project is fundamental. I'm glad you are working on it.
I have researched this topic in my PhD thesis and I went through a review of the online communities engagement literature.
Few ideas for metrics:
- Contributions inequality measurements (gini coefficients as a start).
- Multilingual editors contributions (to see whether they see Wikipedia
as a global project or prefer to focus on one language).
- Core-periphery social interactions (admins-newbees, in order to detect
communities more prone to mentoring)
- Rate of newbies completing the first article, rate of newbies
completing the first translation, etc.
- Recency measures for newbies (different measures on editor retention).
- Community/functional roles renewal (admin, autopatrolled, etc. to see
how good a community is at renewing its core along the years).
I'd be happy to further discuss the topic. At your disposal. Best regards,
Marc Miquel
El dv., 5 d’oct. 2018 a les 23:29, Joe Sutherland (< jsutherland@wikimedia.org>) va escriure:
Hello everyone - apologies for cross-posting! *TL;DR*: We would like your feedback on our Metrics Kit project. Please have a look and comment on Meta-Wiki: https://meta.wikimedia.org/wiki/Community_health_initiative/Metrics_kit
The Wikimedia Foundation's Trust and Safety team, in collaboration with the Community Health Initiative, is working on a Metrics Kit designed to measure the relative "health"[1] of various communities that make up the Wikimedia movement: https://meta.wikimedia.org/wiki/Community_health_initiative/Metrics_kit
The ultimate outcome will be a public suite of statistics and data looking at various aspects of Wikimedia project communities. This could be used by both community members to make decisions on their community direction and Wikimedia Foundation staff to point anti-harassment tool development in the right direction.
We have a set of metrics we are thinking about including in the kit, ranging from the ratio of active users to active administrators, administrator confidence levels, and off-wiki factors such as freedom to participate. It's ambitious, and our methods of collecting such data will vary.
Right now, we'd like to know:
- Which metrics make sense to collect? Which don't? What are we missing?
- Where would such a tool ideally be hosted? Where would you normally
look for statistics like these?
- We are aware of the overlap in scope between this and Wikistats <
https://stats.wikimedia.org/v2/#/all-projects%3E — how might these tools coexist?
Your opinions will help to guide this project going forward. We'll be reaching out at different stages of this project, so if you're interested in direct messaging going forward, please feel free to indicate your interest by signing up on the consultation page.
Looking forward to reading your thoughts.
best, Joe
P.S.: Please feel free to CC me in conversations that might happen on this list!
[1] What do we mean by "health"? There is no standard definition of what makes a Wikimedia community "healthy", but there are many indicators that highlight where a wiki is doing well, and where it could improve. This project aims to provide a variety of useful data points that will inform community decisions that will benefit from objective data.
-- *Joe Sutherland* (he/him or they/them) Trust and Safety Specialist Wikimedia Foundation joesutherland.rocks _______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- Saludos, Abel. _______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Hey Joe, whatever happened with this? Is it still being worked on?
On Fri, Oct 5, 2018 at 3:29 PM Joe Sutherland jsutherland@wikimedia.org wrote:
Hello everyone - apologies for cross-posting! *TL;DR*: We would like your feedback on our Metrics Kit project. Please have a look and comment on Meta-Wiki: https://meta.wikimedia.org/wiki/Community_health_initiative/Metrics_kit
The Wikimedia Foundation's Trust and Safety team, in collaboration with the Community Health Initiative, is working on a Metrics Kit designed to measure the relative "health"[1] of various communities that make up the Wikimedia movement: https://meta.wikimedia.org/wiki/Community_health_initiative/Metrics_kit
The ultimate outcome will be a public suite of statistics and data looking at various aspects of Wikimedia project communities. This could be used by both community members to make decisions on their community direction and Wikimedia Foundation staff to point anti-harassment tool development in the right direction.
We have a set of metrics we are thinking about including in the kit, ranging from the ratio of active users to active administrators, administrator confidence levels, and off-wiki factors such as freedom to participate. It's ambitious, and our methods of collecting such data will vary.
Right now, we'd like to know:
- Which metrics make sense to collect? Which don't? What are we missing?
- Where would such a tool ideally be hosted? Where would you normally look
for statistics like these?
- We are aware of the overlap in scope between this and Wikistats <
https://stats.wikimedia.org/v2/#/all-projects%3E — how might these tools coexist?
Your opinions will help to guide this project going forward. We'll be reaching out at different stages of this project, so if you're interested in direct messaging going forward, please feel free to indicate your interest by signing up on the consultation page.
Looking forward to reading your thoughts.
best, Joe
P.S.: Please feel free to CC me in conversations that might happen on this list!
[1] What do we mean by "health"? There is no standard definition of what makes a Wikimedia community "healthy", but there are many indicators that highlight where a wiki is doing well, and where it could improve. This project aims to provide a variety of useful data points that will inform community decisions that will benefit from objective data.
-- *Joe Sutherland* (he/him or they/them) Trust and Safety Specialist Wikimedia Foundation joesutherland.rocks _______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Anyone else know anything about the fate of the community health metrics kit? The wiki page https://meta.wikimedia.org/wiki/Community_health_initiative/Metrics_kit still says it's expected to be launched in 2019.
On Thu, Feb 20, 2020 at 1:19 PM Ryan Kaldari rkaldari@wikimedia.org wrote:
Hey Joe, whatever happened with this? Is it still being worked on?
On Fri, Oct 5, 2018 at 3:29 PM Joe Sutherland jsutherland@wikimedia.org wrote:
Hello everyone - apologies for cross-posting! *TL;DR*: We would like your feedback on our Metrics Kit project. Please have a look and comment on Meta-Wiki: https://meta.wikimedia.org/wiki/Community_health_initiative/Metrics_kit
The Wikimedia Foundation's Trust and Safety team, in collaboration with the Community Health Initiative, is working on a Metrics Kit designed to measure the relative "health"[1] of various communities that make up the Wikimedia movement: https://meta.wikimedia.org/wiki/Community_health_initiative/Metrics_kit
The ultimate outcome will be a public suite of statistics and data looking at various aspects of Wikimedia project communities. This could be used by both community members to make decisions on their community direction and Wikimedia Foundation staff to point anti-harassment tool development in the right direction.
We have a set of metrics we are thinking about including in the kit, ranging from the ratio of active users to active administrators, administrator confidence levels, and off-wiki factors such as freedom to participate. It's ambitious, and our methods of collecting such data will vary.
Right now, we'd like to know:
- Which metrics make sense to collect? Which don't? What are we missing?
- Where would such a tool ideally be hosted? Where would you normally
look for statistics like these?
- We are aware of the overlap in scope between this and Wikistats <
https://stats.wikimedia.org/v2/#/all-projects%3E — how might these tools coexist?
Your opinions will help to guide this project going forward. We'll be reaching out at different stages of this project, so if you're interested in direct messaging going forward, please feel free to indicate your interest by signing up on the consultation page.
Looking forward to reading your thoughts.
best, Joe
P.S.: Please feel free to CC me in conversations that might happen on this list!
[1] What do we mean by "health"? There is no standard definition of what makes a Wikimedia community "healthy", but there are many indicators that highlight where a wiki is doing well, and where it could improve. This project aims to provide a variety of useful data points that will inform community decisions that will benefit from objective data.
-- *Joe Sutherland* (he/him or they/them) Trust and Safety Specialist Wikimedia Foundation joesutherland.rocks _______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
In my last meeting with Joe they were still collecting requirements, but that was before Joe got shifted around the org and I'm guessing the project is now done.
This effort is exactly what I'm talking about in my email to product-all and tech-all yesterday (subject: The Question and Answer Flow). Here we have a set of questions about community health that we've stopped asking because it hasn't been possible to get answers. I suggest folks who care about this chime in there and point to how these metrics would help them.
On Tue, Feb 25, 2020 at 12:15 PM Ryan Kaldari rkaldari@wikimedia.org wrote:
Anyone else know anything about the fate of the community health metrics kit? The wiki page https://meta.wikimedia.org/wiki/Community_health_initiative/Metrics_kit still says it's expected to be launched in 2019.
On Thu, Feb 20, 2020 at 1:19 PM Ryan Kaldari rkaldari@wikimedia.org wrote:
Hey Joe, whatever happened with this? Is it still being worked on?
On Fri, Oct 5, 2018 at 3:29 PM Joe Sutherland jsutherland@wikimedia.org wrote:
Hello everyone - apologies for cross-posting! *TL;DR*: We would like your feedback on our Metrics Kit project. Please have a look and comment on Meta-Wiki: https://meta.wikimedia.org/wiki/Community_health_initiative/Metrics_kit
The Wikimedia Foundation's Trust and Safety team, in collaboration with the Community Health Initiative, is working on a Metrics Kit designed to measure the relative "health"[1] of various communities that make up the Wikimedia movement: https://meta.wikimedia.org/wiki/Community_health_initiative/Metrics_kit
The ultimate outcome will be a public suite of statistics and data looking at various aspects of Wikimedia project communities. This could be used by both community members to make decisions on their community direction and Wikimedia Foundation staff to point anti-harassment tool development in the right direction.
We have a set of metrics we are thinking about including in the kit, ranging from the ratio of active users to active administrators, administrator confidence levels, and off-wiki factors such as freedom to participate. It's ambitious, and our methods of collecting such data will vary.
Right now, we'd like to know:
- Which metrics make sense to collect? Which don't? What are we missing?
- Where would such a tool ideally be hosted? Where would you normally
look for statistics like these?
- We are aware of the overlap in scope between this and Wikistats <
https://stats.wikimedia.org/v2/#/all-projects%3E — how might these tools coexist?
Your opinions will help to guide this project going forward. We'll be reaching out at different stages of this project, so if you're interested in direct messaging going forward, please feel free to indicate your interest by signing up on the consultation page.
Looking forward to reading your thoughts.
best, Joe
P.S.: Please feel free to CC me in conversations that might happen on this list!
[1] What do we mean by "health"? There is no standard definition of what makes a Wikimedia community "healthy", but there are many indicators that highlight where a wiki is doing well, and where it could improve. This project aims to provide a variety of useful data points that will inform community decisions that will benefit from objective data.
-- *Joe Sutherland* (he/him or they/them) Trust and Safety Specialist Wikimedia Foundation joesutherland.rocks _______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Hi,
On 25/2/20 18:35, Dan Andreescu wrote:
In my last meeting with Joe they were still collecting requirements, but that was before Joe got shifted around the org and I'm guessing the project is now done.
This effort is exactly what I'm talking about in my email to product-all and tech-all yesterday (subject: The Question and Answer Flow). Here we have a set of questions about community health that we've stopped asking because it hasn't been possible to get answers. I suggest folks who care about this chime in there and point to how these metrics would help them.
I wanted to share this project that I have just proposed for a project grant:
https://meta.wikimedia.org/wiki/Grants:Project/Community_Health_Metrics:_Und...
We are team of 3 researchers from Eurecat, a research center in Barcelona. We have quite an extensive experience doing research on Wikipedia both on content and user interactions.
We understand that the project is quite ambitious and we would like to involve the community and people that have already worked on the topic (for example, we have asked Aaron Halfaker to be an advisor for the project).
For this reason, we would love to have the feedback from this list, and especially yours, Dan.
We want to make a prototype dashboard as an output of the project so that the community can see the results of our metrics. To be fair, let me stress the fact that we want to build a /prototype/. We are researcher, our main expertise lies in the metrics and analyses. Also, we are interested in studying editor drop-off, so the metrics that we will focus on will be the one that we find are relevant with respect to that.
I hope that you could find this project useful and I hope to read your feedback.
Cristian
A quick update: I talked to Patrick Earley and he informed me that the Community Health Metrics Kit project was put on ice last year due to lack of resources.
On Wed, Feb 26, 2020 at 4:57 AM Cristian Consonni cristian@balist.es wrote:
Hi,
On 25/2/20 18:35, Dan Andreescu wrote:
In my last meeting with Joe they were still collecting requirements, but that was before Joe got shifted around the org and I'm guessing the project is now done.
This effort is exactly what I'm talking about in my email to product-all and tech-all yesterday (subject: The Question and Answer Flow). Here we have a set of questions about community health that we've stopped asking because it hasn't been possible to get answers. I suggest folks who care about this chime in there and point to how these metrics would help them.
I wanted to share this project that I have just proposed for a project grant:
https://meta.wikimedia.org/wiki/Grants:Project/Community_Health_Metrics:_Und...
We are team of 3 researchers from Eurecat, a research center in Barcelona. We have quite an extensive experience doing research on Wikipedia both on content and user interactions.
We understand that the project is quite ambitious and we would like to involve the community and people that have already worked on the topic (for example, we have asked Aaron Halfaker to be an advisor for the project).
For this reason, we would love to have the feedback from this list, and especially yours, Dan.
We want to make a prototype dashboard as an output of the project so that the community can see the results of our metrics. To be fair, let me stress the fact that we want to build a /prototype/. We are researcher, our main expertise lies in the metrics and analyses. Also, we are interested in studying editor drop-off, so the metrics that we will focus on will be the one that we find are relevant with respect to that.
I hope that you could find this project useful and I hope to read your feedback.
Cristian
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics