As many of you are aware, Discovery wants to run a QuickSurvey https://www.mediawiki.org/wiki/Extension:QuickSurveys in Q3 to ask users if they're satisfied with search results. A requirement of this is that we can tie the survey responses to our search schema and satisfaction metric, so that we can correlate responses with the data to figure out how effective our metric actually is at measuring search satisfaction.
Adam, Julien, and I had a brief chat today. We agreed that our goal is to be able to tie the data together by whatever means necessary, i.e. not necessarily by changing QuickSurveys if it's easier a different way. Adam mentioned that QuickSurveys records mw.user.sessionId, which may be suitable and persistent enough that we could tie our data together if we added that to our search logging. Obviously, there are other stakeholders to talk to (Erik, Oliver) and questions to resolve; Julien wants to have a meeting with Erik and Oliver next week.
Overall, this is good news. :-)
Thanks!
Dan
Sounds good! I'm happy to join in the conversation as well!
Julien, if you've scheduled the meeting already, can you invite me? Otherwise, I can set one up for next week with the interested parties. :)
Cheers,
Deb
-- Deb Tankersley Product Manager, Discovery Wikimedia Foundation
On Fri, Jan 22, 2016 at 6:06 PM, Dan Garry dgarry@wikimedia.org wrote:
As many of you are aware, Discovery wants to run a QuickSurvey https://www.mediawiki.org/wiki/Extension:QuickSurveys in Q3 to ask users if they're satisfied with search results. A requirement of this is that we can tie the survey responses to our search schema and satisfaction metric, so that we can correlate responses with the data to figure out how effective our metric actually is at measuring search satisfaction.
Adam, Julien, and I had a brief chat today. We agreed that our goal is to be able to tie the data together by whatever means necessary, i.e. not necessarily by changing QuickSurveys if it's easier a different way. Adam mentioned that QuickSurveys records mw.user.sessionId, which may be suitable and persistent enough that we could tie our data together if we added that to our search logging. Obviously, there are other stakeholders to talk to (Erik, Oliver) and questions to resolve; Julien wants to have a meeting with Erik and Oliver next week.
Overall, this is good news. :-)
Thanks!
Dan
-- Dan Garry Lead Product Manager, Discovery Wikimedia Foundation
discovery mailing list discovery@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/discovery
I'm glad this is happening. Happy to join this too if you need me.
On Fri, Jan 22, 2016 at 5:29 PM, Deborah Tankersley < dtankersley@wikimedia.org> wrote:
Sounds good! I'm happy to join in the conversation as well!
Julien, if you've scheduled the meeting already, can you invite me? Otherwise, I can set one up for next week with the interested parties. :)
Cheers,
Deb
-- Deb Tankersley Product Manager, Discovery Wikimedia Foundation
On Fri, Jan 22, 2016 at 6:06 PM, Dan Garry dgarry@wikimedia.org wrote:
As many of you are aware, Discovery wants to run a QuickSurvey https://www.mediawiki.org/wiki/Extension:QuickSurveys in Q3 to ask users if they're satisfied with search results. A requirement of this is that we can tie the survey responses to our search schema and satisfaction metric, so that we can correlate responses with the data to figure out how effective our metric actually is at measuring search satisfaction.
Adam, Julien, and I had a brief chat today. We agreed that our goal is to be able to tie the data together by whatever means necessary, i.e. not necessarily by changing QuickSurveys if it's easier a different way. Adam mentioned that QuickSurveys records mw.user.sessionId, which may be suitable and persistent enough that we could tie our data together if we added that to our search logging. Obviously, there are other stakeholders to talk to (Erik, Oliver) and questions to resolve; Julien wants to have a meeting with Erik and Oliver next week.
Overall, this is good news. :-)
Thanks!
Dan
-- Dan Garry Lead Product Manager, Discovery Wikimedia Foundation
discovery mailing list discovery@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/discovery
discovery mailing list discovery@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/discovery
The tasks to track for the forthcoming enhancement on the QS extension that has this mw.user.sessionid bearing value (it has "-quicksurvey" appended to it, by the way) is in the extension deployment task https://phabricator.wikimedia.org/T124488 and its child task for the code level changes at https://phabricator.wikimedia.org/T123696. We're in the middle of sprint 64, and the parent deployment task is in the prioritized backlog tentatively for sprint 65.
-Adam
On Friday, January 22, 2016, Dan Garry dgarry@wikimedia.org wrote:
As many of you are aware, Discovery wants to run a QuickSurvey https://www.mediawiki.org/wiki/Extension:QuickSurveys in Q3 to ask users if they're satisfied with search results. A requirement of this is that we can tie the survey responses to our search schema and satisfaction metric, so that we can correlate responses with the data to figure out how effective our metric actually is at measuring search satisfaction.
Adam, Julien, and I had a brief chat today. We agreed that our goal is to be able to tie the data together by whatever means necessary, i.e. not necessarily by changing QuickSurveys if it's easier a different way. Adam mentioned that QuickSurveys records mw.user.sessionId, which may be suitable and persistent enough that we could tie our data together if we added that to our search logging. Obviously, there are other stakeholders to talk to (Erik, Oliver) and questions to resolve; Julien wants to have a meeting with Erik and Oliver next week.
Overall, this is good news. :-)
Thanks!
Dan
-- Dan Garry Lead Product Manager, Discovery Wikimedia Foundation
On Sat, Jan 23, 2016 at 12:06 PM, Dan Garry dgarry@wikimedia.org wrote:
As many of you are aware, Discovery wants to run a QuickSurvey in Q3 to ask users if they're satisfied with search results. A requirement of this is that we can tie the survey responses to our search schema and satisfaction metric, so that we can correlate responses with the data to figure out how effective our metric actually is at measuring search satisfaction.
I hope that this isn't a metric in isolation. I would have thought that metrics like * immediate re-search (refinement) * no clicking / following of a search result (resignation) would be of more relevance. Asking such a question of someone who did NOT get a positive search result and didn't follow a link will have an obvious answer, so asking for their satisfaction would almost be self-evident. One might also think that a survey response from those who had a failed search is going to be more likely to occur (and in expected direction), rather than a response from someone who had success, and went to the link. So do we have information on the bias response in surveys in such circumstances?
Now maybe we have a different interpretation of the word "satisfaction" but that all seems to be with regard to the warm inner glow of finding something, rather than any of the technical aspects.
Also, we have to presume that in a "satisfaction" survey that we are able to differentiate between no satisfaction for zero results, when we are not having information on the subject, compared to when it is no result for where search failed to find something that we do have information.
OR are you intentionally focusing on zero resulters, no followers, or on those disatisified with their landing page on a GO result (ie. the GO results, rather than a search result?)
Where are you intending to run these surveys? Languages? Sisters? How will such results be compiled collectively? Or split through the wikis? Will these results be available to the communities?
[snip]
-- Dan Garry Lead Product Manager, Discovery Wikimedia Foundation
-- billinghurst
Hey Billinghurst,
Thanks for the reply! Responses in-line.
On 22 January 2016 at 18:42, billinghurst billinghurstwiki@gmail.com wrote:
I hope that this isn't a metric in isolation. I would have thought that metrics like
- immediate re-search (refinement)
- no clicking / following of a search result (resignation)
would be of more relevance.
We track both of these metrics already, and don't have much need to iterate on them because they're relatively clear-cut in terms of the implementation and how the data is collected. In fact, the search satisfaction metric relies explicitly on the user clicking on a result, so that's not only tracked, but also factored directly in to that metric.
Asking such a question of someone who did NOT get a positive search result and didn't follow a link will have an obvious answer, so asking for their satisfaction would almost be self-evident. One might also think that a survey response from those who had a failed search is going to be more likely to occur (and in expected direction), rather than a response from someone who had success, and went to the link.
Is going to a link a success? Not necessarily, if you find out that the link you went to is the wrong link, or is irrelevant for some reason. That's what we're aiming to find out. :-)
Now maybe we have a different interpretation of the word "satisfaction" but that all seems to be with regard to the warm inner glow of finding something, rather than any of the technical aspects.
That's intentional. The name was chosen because we started first with what we wanted to measure ("user satisfaction with search"), then worked backwards until we found a technical solution that worked.
Also, we have to presume that in a "satisfaction" survey that we are able to differentiate between no satisfaction for zero results, when we are not having information on the subject, compared to when it is no result for where search failed to find something that we do have information.
Yes, we measure the overall zero results rate http://discovery.wmflabs.org/metrics/#kpi_zero_results already, so we already know what portion of queries are unsatisfying there. That needs little further research in this context. What we're trying to figure out here is how satisfied are people when they do have results; are all the results irrelevant, or are some useful?
OR are you intentionally focusing on zero resulters, no followers, or on those disatisified with their landing page on a GO result (ie. the GO results, rather than a search result?)
We're focussing specifically on any user who types in a query then goes a page with this specific metric. We are also implicitly counting users who type in queries and then do *not* click on a result, as those are counted as unsatisfied users.
Where are you intending to run these surveys? Languages? Sisters?
I don't know what scope we'll go for at first. Ideally, we'll run it on all wikis, but if we have to cut it down to a smaller set in order to keep the work well scoped, then we will.
How will such results be compiled collectively? Or split through the wikis?
Well, the wiki the user is on will be recorded, so splitting things up will be fairly straightforwards.
Will these results be available to the communities?
Of course. Discovery publishes public reports of all of our analyses, and we maintain public dashboards https://discovery.wmflabs.org for all of our focus areas; this will be no different. :-)
Hopefully that helps bring some clarification.
Thanks, Dan
Will reply to the full body in time.
Just want to note that https://discovery.wmflabs.org/metrics/ is causing errors for me with Chrome and Firefox.
" The application unexpectedly exited.
Diagnostic information has been dumped to the JavaScript error console "
Diagnostic errors from consoles are not my zone. :-/
-- billinghurst
On 22 January 2016 at 19:10, billinghurst billinghurstwiki@gmail.com wrote:
Just want to note that https://discovery.wmflabs.org/metrics/ is causing errors for me with Chrome and Firefox.
" The application unexpectedly exited.
Diagnostic information has been dumped to the JavaScript error console "
Diagnostic errors from consoles are not my zone. :-/
Thanks for the report. I'm not seeing this error myself; the dashboard works fine for me. If you refresh, does the problem persist? Regardless, we will investigate a bit next week.
Thanks, Dan
Took two purges of Chrome to make the error go away.
For Firefox, I had to get a browser update, I am on 64-bit beta, so seems that the issue was local. Just weird to have different browsers causing issues. <shrug>
Thanks.
On Sat, Jan 23, 2016 at 2:12 PM, Dan Garry dgarry@wikimedia.org wrote:
On 22 January 2016 at 19:10, billinghurst billinghurstwiki@gmail.com wrote:
Just want to note that https://discovery.wmflabs.org/metrics/ is causing errors for me with Chrome and Firefox.
" The application unexpectedly exited.
Diagnostic information has been dumped to the JavaScript error console "
Diagnostic errors from consoles are not my zone. :-/
Thanks for the report. I'm not seeing this error myself; the dashboard works fine for me. If you refresh, does the problem persist? Regardless, we will investigate a bit next week.
Thanks, Dan
-- Dan Garry Lead Product Manager, Discovery Wikimedia Foundation
discovery mailing list discovery@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/discovery
Hi there - what versions of Chrome and FF are you using and what OS do you have running? Knowing that info will help us test out things here...
Cheers,
Deb
-- Deb Tankersley Product Manager, Discovery Wikimedia Foundation
On Fri, Jan 22, 2016 at 8:31 PM, billinghurst billinghurstwiki@gmail.com wrote:
Took two purges of Chrome to make the error go away.
For Firefox, I had to get a browser update, I am on 64-bit beta, so seems that the issue was local. Just weird to have different browsers causing issues. <shrug>
Thanks.
On Sat, Jan 23, 2016 at 2:12 PM, Dan Garry dgarry@wikimedia.org wrote:
On 22 January 2016 at 19:10, billinghurst billinghurstwiki@gmail.com wrote:
Just want to note that https://discovery.wmflabs.org/metrics/ is causing errors for me with Chrome and Firefox.
" The application unexpectedly exited.
Diagnostic information has been dumped to the JavaScript error console "
Diagnostic errors from consoles are not my zone. :-/
Thanks for the report. I'm not seeing this error myself; the dashboard
works
fine for me. If you refresh, does the problem persist? Regardless, we
will
investigate a bit next week.
Thanks, Dan
-- Dan Garry Lead Product Manager, Discovery Wikimedia Foundation
discovery mailing list discovery@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/discovery
discovery mailing list discovery@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/discovery