We've now got some preliminary data in about how effective our search is in mobile apps. Note that this is only half a day's data, and it's sampled at 1%.

High-level metrics:
  • Number of searching sessions which end by tapping on a result: 91%
  • Percentage of queries that give 0 results: 19%
  • Mean queries per searching session: 3.8
  • Median user-perceived time taken to retrieve search results: 486ms
  • 90th percentile of user-perceived time taken: 898ms
My quick take-homes from this:
  • 91% clickthrough seems pretty good.
  • 19% of queries giving "no results" seems really bad.
  • The combination of the above two means users are generally finding what they need, but that it's a struggle to do it.
  • User-perceived performance seems pretty good; 90% of our users get search results within a second!
What I think our next steps are:
  • Push out our new search improvements and see how these baselines change.
  • Try to tackle the "0 results" problem, which seems to be more of a problem than clickthrough.
Spreadsheet containing my queries and processed data: https://docs.google.com/a/wikimedia.org/spreadsheets/d/1syLgNygAS7Prxxg7RTIvK3vxwyVG-ZkjxtK0LSbU76w/edit

Dan

--
Dan Garry
Associate Product Manager, Mobile Apps
Wikimedia Foundation