New subject: Preliminary WikiGrok response quality in stable

5 Jan 2015

If you're like me, you've probably been breathlessly awaiting the results
of the first WikiGrok stable A/B test to see if the responses we're getting
are good, bad, or ugly :) Well, good news! I did some hand-coding of the
results (a sample of about 300 responses from the ~1,200 we got during the
test) and have some interesting preliminary findings to share. Caveat: this
is not science, just a quick check of WikiGrok's pulse. Leila from
Analytics is helping us analyze this and other WikiGrok test data and will
have a more thorough write-up of the results soon :)

As a reminder, this test ran for a week in December in stable for logged in
users only on English Wikipedia. We tested two versions of the UX (a simple
"yes/no/maybe" interface and a slightly more complex tagging one), and we
asked questions about biographies (actors and writers) and music albums
(live or studio albums). The responses were not yet sent to Wikidata; the
infrastructure to do that is currently in development.

* The tl;dr is that the quality of the responses is pretty high! *The
overall rate of correct responses for the sample I looked at was* *80%*.

* Also, *users with no edits and users with 1 or more edits had similar
quality responses* (in fact, the 0 edit count users gave slightly higher
quality responses). So even total newbs are capable of grokking :)

* Lastly, while we didn't see any differences in engagement or conversion
(the rate at which users started and finished the WikiGrok process) between
the two versions, there was a difference in quality – *Version B (tagging)
produced a noticeably higher quality response rate (95%)*.

More detailed breakdown of quality below, including by individual answer
(fun fact that is sure to make Sam Smith sad: nobody seems to have any clue
what a live album is!). Now let's see if these trends hold for logged out
users, too :) Our first test for all users (logged in and logged out) is
slated for later this month.

** * * *

*User classes*

Users with 0 edits – 85%
Users with 1 or more edits – 80%

*Versions*

Version A – 68%
Version B – 95%

*Question types*

"Is this person an author?" – 72%
"Is this a film actor?" – 90%
"Is this a television actor?" – 85%
"Is this a live album?" – 50% :(
"Is this a studio album?" – 64%

-- 
Maryana Pinchuk
Product Manager, Wikimedia Foundation
wikimediafoundation.org