Hey Billinghurst,
Thanks for the reply! Responses in-line.
On 22 January 2016 at 18:42, billinghurst billinghurstwiki@gmail.com wrote:
I hope that this isn't a metric in isolation. I would have thought that metrics like
- immediate re-search (refinement)
- no clicking / following of a search result (resignation)
would be of more relevance.
We track both of these metrics already, and don't have much need to iterate on them because they're relatively clear-cut in terms of the implementation and how the data is collected. In fact, the search satisfaction metric relies explicitly on the user clicking on a result, so that's not only tracked, but also factored directly in to that metric.
Asking such a question of someone who did NOT get a positive search result and didn't follow a link will have an obvious answer, so asking for their satisfaction would almost be self-evident. One might also think that a survey response from those who had a failed search is going to be more likely to occur (and in expected direction), rather than a response from someone who had success, and went to the link.
Is going to a link a success? Not necessarily, if you find out that the link you went to is the wrong link, or is irrelevant for some reason. That's what we're aiming to find out. :-)
Now maybe we have a different interpretation of the word "satisfaction" but that all seems to be with regard to the warm inner glow of finding something, rather than any of the technical aspects.
That's intentional. The name was chosen because we started first with what we wanted to measure ("user satisfaction with search"), then worked backwards until we found a technical solution that worked.
Also, we have to presume that in a "satisfaction" survey that we are able to differentiate between no satisfaction for zero results, when we are not having information on the subject, compared to when it is no result for where search failed to find something that we do have information.
Yes, we measure the overall zero results rate http://discovery.wmflabs.org/metrics/#kpi_zero_results already, so we already know what portion of queries are unsatisfying there. That needs little further research in this context. What we're trying to figure out here is how satisfied are people when they do have results; are all the results irrelevant, or are some useful?
OR are you intentionally focusing on zero resulters, no followers, or on those disatisified with their landing page on a GO result (ie. the GO results, rather than a search result?)
We're focussing specifically on any user who types in a query then goes a page with this specific metric. We are also implicitly counting users who type in queries and then do *not* click on a result, as those are counted as unsatisfied users.
Where are you intending to run these surveys? Languages? Sisters?
I don't know what scope we'll go for at first. Ideally, we'll run it on all wikis, but if we have to cut it down to a smaller set in order to keep the work well scoped, then we will.
How will such results be compiled collectively? Or split through the wikis?
Well, the wiki the user is on will be recorded, so splitting things up will be fairly straightforwards.
Will these results be available to the communities?
Of course. Discovery publishes public reports of all of our analyses, and we maintain public dashboards https://discovery.wmflabs.org for all of our focus areas; this will be no different. :-)
Hopefully that helps bring some clarification.
Thanks, Dan