I took a look at the early data from the WikiGrok prototypes. None of this data is being posted to Wikidata yet since we want to get an idea of the quality level before we do anything with it. The bad news is we have very little data so far, the good news is that the response we do have are very high quality.

We collected a total of 229 responses. About half of them were from a short-lived bug that showed WikiGrok on all articles (for logged in Beta users), a quarter were internal tests, and 45 were legitimate user responses on appropriate articles. All 45 of the legitimate responses were from WikiGrok version A (which is the version in Beta).

Of the 45 responses, 42 were definitely answered correctly. The other 3 were ambiguous, i.e. it isn't 100% clear what the correct answer is. As an example of an ambiguous case, a user marked the following claim as true: 'Kailash Satyarthi: occupation->engineer'. The article states that Kailash Satyarthi studied engineering in college, and may have taught engineering, although it is unclear if he ever practiced engineering. The article is mainly about his career as a political activist.

The most interesting thing about the responses so far is that there are zero blatantly wrong claims. In other words, it appears that all of the users so far have at least tried to answer the questions in good faith.

Only 9 of the 45 claims are anti-claims, i.e. 'John Doe was not a teacher'. It isn't clear, however, if this is due to the fact that negative claims are much harder to be certain about or due to most of the potential claims in the suggestions database being accurate claims.

Looking at the data from the bug period (Oct. 16) is also interesting. Due to an API bug, we temporarily surfaced WikiGrok on all articles. This resulted in nonsensical questions like "Was Supersymmetry a politician?" Even in the face of nonsense questions, 94% of responses were accurate.