We've now got some preliminary data in about how effective our search is in mobile apps. Note that this is only half a day's data, and it's sampled at 1%.
High-level metrics:
- *Number of searching sessions which end by tapping on a result:* 91% - *Percentage of queries that give 0 results: *19% - *Mean queries per searching session:* 3.8 - *Median user-perceived time taken to retrieve search results: *486ms - *90th percentile of user-perceived time taken:* 898ms
My quick take-homes from this:
- 91% clickthrough seems pretty good. - 19% of queries giving "no results" seems really bad. - The combination of the above two means users are generally finding what they need, but that it's a struggle to do it. - User-perceived performance seems pretty good; 90% of our users get search results within a second!
What I think our next steps are:
- Push out our new search improvements and see how these baselines change. - Try to tackle the "0 results" problem, which seems to be more of a problem than clickthrough.
Spreadsheet containing my queries and processed data: https://docs.google.com/a/wikimedia.org/spreadsheets/d/1syLgNygAS7Prxxg7RTIv...
Dan
On Wed, Dec 3, 2014 at 6:15 PM, Dan Garry dgarry@wikimedia.org wrote:
19% of queries giving "no results" seems really bad.
Fixing this needs to be our priority across all projects. It doesn't matter what we fix later down the pipeline if this number is as high as it is. Let's get this as a daily/hourly number so that we can see live stats and then discuss changes/improvements.
Great work getting these --tomasz
Since presently all app searches are prefix, I'm not too surprised by 19% - I seem to misspell at least that percentage typing on phone screens, and most misspellings result in zero prefix results. But yes, really excited about supplementing small/null prefix results sets with full text results as suggested by Jared. We'll see that 19% really drop.
On Dec 3, 2014, at 6:27 PM, Tomasz Finc tfinc@wikimedia.org wrote:
On Wed, Dec 3, 2014 at 6:15 PM, Dan Garry dgarry@wikimedia.org wrote: 19% of queries giving "no results" seems really bad.
Fixing this needs to be our priority across all projects. It doesn't matter what we fix later down the pipeline if this number is as high as it is. Let's get this as a daily/hourly number so that we can see live stats and then discuss changes/improvements.
Great work getting these --tomasz
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Nemo,
Prefix search doesn't offer suggestions. So, 0 then.
Bernd
On Wed, Dec 3, 2014 at 11:28 PM, Federico Leva (Nemo) nemowiki@gmail.com wrote:
Dan Garry, 04/12/2014 03:15:
# *Percentage of queries that give 0 results: *19%
How many of these offered a "did you mean" alternative? Can typos on mobile increase the error rate?
Nemo
P.s.: Someone add this number next to the low-hanging fruit hypothesis claims. ;-)
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Thanks.
This is not directly related, but some time I'd love to see data about interlanguage links tapping on all media - iOS, Android, MobileFrontend.
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore
2014-12-04 4:15 GMT+02:00 Dan Garry dgarry@wikimedia.org:
We've now got some preliminary data in about how effective our search is in mobile apps. Note that this is only half a day's data, and it's sampled at 1%.
High-level metrics:
- *Number of searching sessions which end by tapping on a result:* 91%
- *Percentage of queries that give 0 results: *19%
- *Mean queries per searching session:* 3.8
- *Median user-perceived time taken to retrieve search results: *486ms
- *90th percentile of user-perceived time taken:* 898ms
My quick take-homes from this:
- 91% clickthrough seems pretty good.
- 19% of queries giving "no results" seems really bad.
- The combination of the above two means users are generally finding
what they need, but that it's a struggle to do it.
- User-perceived performance seems pretty good; 90% of our users get
search results within a second!
What I think our next steps are:
- Push out our new search improvements and see how these baselines
change.
- Try to tackle the "0 results" problem, which seems to be more of a
problem than clickthrough.
Spreadsheet containing my queries and processed data: https://docs.google.com/a/wikimedia.org/spreadsheets/d/1syLgNygAS7Prxxg7RTIv...
Dan
-- Dan Garry Associate Product Manager, Mobile Apps Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
I noticed many bilingual friends search in their native language on English Wikipedia. It would be interesting to look closely at search terms and see if they are mispellings, different languages or just our search failing them.
On Wed, Dec 3, 2014 at 11:01 PM, Amir E. Aharoni amir.aharoni@mail.huji.ac.il wrote:
Thanks.
This is not directly related, but some time I'd love to see data about interlanguage links tapping on all media - iOS, Android, MobileFrontend.
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore
2014-12-04 4:15 GMT+02:00 Dan Garry dgarry@wikimedia.org:
We've now got some preliminary data in about how effective our search is in mobile apps. Note that this is only half a day's data, and it's sampled at 1%.
High-level metrics:
Number of searching sessions which end by tapping on a result: 91% Percentage of queries that give 0 results: 19% Mean queries per searching session: 3.8 Median user-perceived time taken to retrieve search results: 486ms 90th percentile of user-perceived time taken: 898ms
My quick take-homes from this:
91% clickthrough seems pretty good. 19% of queries giving "no results" seems really bad. The combination of the above two means users are generally finding what they need, but that it's a struggle to do it. User-perceived performance seems pretty good; 90% of our users get search results within a second!
What I think our next steps are:
Push out our new search improvements and see how these baselines change. Try to tackle the "0 results" problem, which seems to be more of a problem than clickthrough.
Spreadsheet containing my queries and processed data: https://docs.google.com/a/wikimedia.org/spreadsheets/d/1syLgNygAS7Prxxg7RTIv...
Dan
-- Dan Garry Associate Product Manager, Mobile Apps Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l