Thanks for this great info, Mikhail, and for the work that went into producing this useful - and beautiful! - report.

We do indeed have plans to experiment with the thresholds to find the optimum balance point between wide adoption and maintaining contribution quality, to extend the feature with new microcontribution types, and to begin intensive testing to find ways to encourage users to explore more of the feature, and to inspire long-term use. Advertising new contribution types and re-activation of editors who have previously tried the feature will be one of the mainstays of our push messaging strategy.

If anyone is interested in our particular plans over the next year, don't hesitate to get in touch. We've already been having wider discussions with several teams on how we could possibly work together towards mutual goals.

Cheers, Charlotte

On Fri, May 10, 2019 at 9:37 PM Mikhail Popov mpopov@wikimedia.org wrote:

Hi folks,

I wanted to highlight a few really interesting pieces of data/stats regarding the release of the Suggested Edits feature on Wikipedia app for Android. These come from the daily report https://analytics.wikimedia.org/datasets/wikipedia-android-app-reports/suggested-edits-v1.html, which is also where you'll find a brief description of the feature.

First, at this time 25.3% of editors (whose contributions are being tracked since launch of the backend) have unlocked the feature by making the 5+ title description edits https://www.mediawiki.org/wiki/Wikimedia_Apps/Short_descriptions currently required to unlock it. (See: unlock stats https://analytics.wikimedia.org/datasets/wikipedia-android-app-reports/suggested-edits-v1.html#Unlock-stats) That's 632 editors out of the 2495 editors who have made at least one title description edit since April 5th. We have plans to experiment with this threshold and see what happens if we lower the barrier to entry.

By the way, we don’t expect all logged-in users to edit or unlock the feature (by making the required number of title description edits), as there are incentives on the mobile apps to use an account just for reading (e.g. reading list syncing https://wikimediafoundation.org/2018/05/25/synced-reading-lists/). However, perhaps we should advertise this ability better (especially to logged-in users) and that those title descriptions don’t require any knowledge of wikitext.

And since the production release, the feature has had a steady stream of 20+ users unlocking it per day. What are our users doing with it once they unlock it? They’ve been using it! (Sorry if the text in the included graph is too small to be legible, it's larger in the report.)

Nearly half of all title description edits made with the Android app each day are coming in from editors using the Suggested Edits feature to add & translate descriptions. More than half, even, on some days! Furthermore, some of those edits are made by users who have previously used the feature. Every day we have some editors who are using Suggested Edits for the first time, but there are also quite a few who are returning to the Editor Tasks screen & contributing more. (See: edit stats https://analytics.wikimedia.org/datasets/wikipedia-android-app-reports/suggested-edits-v1.html#Edits-and-editors )

“Okay, so what’s the quality of those 200-400 descriptions being added every day?” you might ask. Well, one way we can check that is to check how many of those edits are reverted within 48 hours. Turns out, almost none of them:

This is especially impressive when compared to the proportion of other title description edits that are reverted. (See: revert rate https://analytics.wikimedia.org/datasets/wikipedia-android-app-reports/suggested-edits-v1.html#Revert-rate )

When the user goes to the Suggested Edits screen and opens a task, they begin receiving suggestions of articles to add descriptions to (or translate descriptions, if they have unlocked that next tier of Suggested Edits). On average, users express interest in editing 30-40% of those suggestions. Among the suggestions they tapped to edit, they end up actually making an edit around 60% of the time (although the average varies from 40% to 70%). (See: interactions https://analytics.wikimedia.org/datasets/wikipedia-android-app-reports/suggested-edits-v1.html#Interaction-with-suggestions and other engagement stats https://analytics.wikimedia.org/datasets/wikipedia-android-app-reports/suggested-edits-v1.html#Usage-stats )

Since the suggested edits are currently completely random, this leaves us with a lot of room for improvement by, say, employing machine learning and simple recommendation systems to suggest articles without title descriptions that are similar to articles the user has added title descriptions to previously. (Just a thought.) For example, in my own experience with the suggestions I tend to skip articles that I don’t feel confident enough to write short descriptions for, which are often articles well outside my interests.

We’re still in the first month of the production release, so it’s hard to draw conclusions about the longevity of this feature. These early numbers are promising, and hopefully the number of editors using this feature continues to grow because then those editors might be inspired to edit articles too (if they haven’t yet). Of course, if we see people get bored over time we might have to consider ways to encourage/inspire long-term use. We also have plans to explore ways to recognize users for their contributions.

So congratulations to the Android & Reading Infrastructure teams and congrats to Rita Ho (now on the Growth team) for an impressive release. We all look forward to the addition of image caption translation and seeing the impact of the expanded Suggested Edits v2 on Structured Data on Commons.

Thanks for reading! :D

Cheers, Mikhail

-- *Mikhail Popov*, Data Analyst (he/him) Product Analytics https://www.mediawiki.org/wiki/Product_Analytics // Audiences https://www.mediawiki.org/wiki/Wikimedia_Audiences // Wikimedia Foundation https://wikimediafoundation.org/

Other info (including PGP): https://people.wikimedia.org/~bearloga/