I took a look at the early data from the WikiGrok prototypes. None of this
data is being posted to Wikidata yet since we want to get an idea of the
quality level before we do anything with it. The bad news is we have very
little data so far, the good news is that the response we do have are very
high quality.
We collected a total of 229 responses. About half of them were from a
short-lived bug that showed WikiGrok on all articles (for logged in Beta
users), a quarter were internal tests, and 45 were legitimate user
responses on appropriate articles. All 45 of the legitimate responses were
from WikiGrok version A (which is the version in Beta).
Of the 45 responses, 42 were definitely answered correctly. The other 3
were ambiguous, i.e. it isn't 100% clear what the correct answer is. As an
example of an ambiguous case, a user marked the following claim as true:
'Kailash Satyarthi: occupation->engineer'. The article states that Kailash
Satyarthi studied engineering in college, and may have taught engineering,
although it is unclear if he ever practiced engineering. The article is
mainly about his career as a political activist.
The most interesting thing about the responses so far is that there are
zero blatantly wrong claims. In other words, it appears that all of the
users so far have at least tried to answer the questions in good faith.
Only 9 of the 45 claims are anti-claims, i.e. 'John Doe was not a teacher'.
It isn't clear, however, if this is due to the fact that negative claims
are much harder to be certain about or due to most of the potential claims
in the suggestions database being accurate claims.
Looking at the data from the bug period (Oct. 16) is also interesting. Due
to an API bug, we temporarily surfaced WikiGrok on all articles. This
resulted in nonsensical questions like "Was Supersymmetry a politician?"
Even in the face of nonsense questions, 94% of responses were accurate.
Kaldari