Dear all,
El lun, 12-08-2013 a las 19:21 +0200, Quim Gil escribió:
你好, 近排點呀? [0]
At Wikimania there was an improvised lunch-meeting about community metrics with Jesús González-Barahona (Bitergia), Sumana, RobLa and myself. The main conclusion was that http://korma.wmflabs.org needs to show a very few key metrics that could drive decisions affecting our strategy and resources.
Yes, it BI terminology, this metrics are the KPI (Key Performance Indicators), so we can reuse this name.
Let's agree first on key factors to watch, in the scope of projects deployed in Wikimedia servers:
- Are the teams more efficient processing contributions?
- Is the share of non-WMF contributions growing?
- Are WMF and non-WMF contributions treated equally?
- Are the attraction and retention of new contributors improving?
- Are we improving the sustainability of our community?
If those are the key factors, these could be the key metrics:
# Who is contributing merged code each quarter? More origins == Better. ## WMF / WMDE / other Wikimedia / companies / OSS projects / independents ## Location of contributors (based on provided data)
# Time to resolve Gerrit changes. Shorter == Better. ## Authored by WMF/WMDE employees vs independent. Should be the same. ## Merged vs rejected. Similar progress? ## Best projects vs bottlenecks.
# Queue of open Gerrit change requests in relation to total amount of contributions. Shorter == Better. ## Same points as above.
# New contributors with 1 / 2-5 / 6+ changes submitted in the past 3 months. More == Better. ## % of merged / rejected. ## Who is growing / stable / vanishing?
# Age of contributors since they started in the project. Small and regular decline == Better. ## Number of contributions from each age group in the past quarters. ## WMF / non-WMF ratio for each age group.
Quim, thinking about those metrics, they are all related to Gerrit and centered around people more than activity.
Any thoughts about the other data sources? From git, the commits metrics is a bit rough, but combining it with SLOC and files, is a good indicator of activity in the development. Also the tendencies for this metrics (week, month and year) are a good indicator of how the project is evolving. For example, last period of 365 days indicators points out that in general, the activity is growing in Wikimedia projects:
http://korma.wmflabs.org/browser/scm.html 63% in commits, 159% in authors and 103% in files.
Taking a look to this metrics, why commits are growing less than authors? Thinking in the metrics and tendencies and relations, we can reach pretty interesting questions to understanding in deep how Wikimedia development is going.
Taking a look to ITS, one metric that I think we are missing is *pending* issues (open issues but not closed). And this concept is something that could be applied to other data sources: the pending work to be done.
Tendencies and pending are two areas in which we can find interesting metrics that could evolve to KPIs.
In ITS we have also the time to close evolution for the 99% of tickets, 95% of them, 50% and 25%. "Time to" is always a good SLA (Service Level Agreement) indicator and at the end, SLA is a strategic position for the community.
In MLS with the current metrics, we can see that around 50% messages generate a thread (has some response), and for example, and interesting metric is that each person send 20 messages (ratio between total messages and senders). So in general, mailing list have feedback and people participate with several messages. Here the "Time to" shows that 95% of messages are attended in less 20 days and that the time to attention has grown last year. The care of the mailing lists is going down?
In demographics we can see pretty good information about all data sources community. Mailing list members are going down also (mailing lists are used less? where is the support of the projects moving on?), SCM new members are also going down but ITS members are growing. The software products start to be mature and more work is centered around quality?
I think Quim that the starting point that you propose is a good one, specially the goal questions. The Goal (improve strategy decision and resources management)/Question/Metrics is the right methodology to follow. With this email I just want to broaden a bit the discussion.
Keep in mind that the deep review of all data in the current Community Dashboard is pending, so take all the numbers above with a grain of salt.
Cheers!
Please have your say. I will be updating http://www.mediawiki.org/wiki/Community_metrics#Key_metrics following the discussion.
[0] http://wikitravel.org/en/Cantonese_phrasebook#Phrase_list ;)