Hey Billinghurst,
Thanks for the reply! Responses in-line.
On 22 January 2016 at 18:42, billinghurst <billinghurstwiki(a)gmail.com>
wrote:
I hope that this isn't a metric in isolation. I would have thought
that metrics like
* immediate re-search (refinement)
* no clicking / following of a search result (resignation)
would be of more relevance.
We track both of these metrics already, and don't have much need to iterate
on them because they're relatively clear-cut in terms of the implementation
and how the data is collected. In fact, the search satisfaction metric
relies explicitly on the user clicking on a result, so that's not only
tracked, but also factored directly in to that metric.
Asking such a question of someone who did
NOT get a positive search result and didn't follow a link will have an
obvious answer, so asking for their satisfaction would almost be
self-evident. One might also think that a survey response from those
who had a failed search is going to be more likely to occur (and in
expected direction), rather than a response from someone who had
success, and went to the link.
Is going to a link a success? Not necessarily, if you find out that the
link you went to is the wrong link, or is irrelevant for some reason.
That's what we're aiming to find out. :-)
Now maybe we have a different interpretation of the
word
"satisfaction" but that all seems to be with regard to the warm inner
glow of finding something, rather than any of the technical aspects.
That's intentional. The name was chosen because we started first with what
we wanted to measure ("user satisfaction with search"), then worked
backwards until we found a technical solution that worked.
Also, we have to presume that in a
"satisfaction" survey that we are
able to differentiate between no satisfaction for zero results, when
we are not having information on the subject, compared to when it is
no result for where search failed to find something that we do have
information.
Yes, we measure the overall zero results rate
<http://discovery.wmflabs.org/metrics/#kpi_zero_results> already, so we
already know what portion of queries are unsatisfying there. That needs
little further research in this context. What we're trying to figure out
here is how satisfied are people when they do have results; are all the
results irrelevant, or are some useful?
OR are you intentionally focusing on zero resulters,
no followers, or
on those disatisified with their landing page on a GO result (ie. the
GO results, rather than a search result?)
We're focussing specifically on any user who types in a query then goes a
page with this specific metric. We are also implicitly counting users who
type in queries and then do *not* click on a result, as those are counted
as unsatisfied users.
Where are you intending to run these surveys?
Languages? Sisters?
I don't know what scope we'll go for at first. Ideally, we'll run it on all
wikis, but if we have to cut it down to a smaller set in order to keep the
work well scoped, then we will.
How will such results be compiled collectively? Or
split through the
wikis?
Well, the wiki the user is on will be recorded, so splitting things up will
be fairly straightforwards.
Will these results be available to the communities?
Of course. Discovery publishes public reports of all of our analyses, and
we maintain public dashboards <https://discovery.wmflabs.org> for all of
our focus areas; this will be no different. :-)
Hopefully that helps bring some clarification.
Thanks,
Dan
--
Dan Garry
Lead Product Manager, Discovery
Wikimedia Foundation