That's actually our goal with quick surveys :P We want to ask users for their satisfaction with our search and then build a predictive model with satisfaction as the response variable and dwell time + other data as the predictor variables.

Right now we're stuck at the "get training data" step. Once that's resolved, we can do precisely what you described :D Then we'll have a daily estimate of user satisfaction (unobservable without direct user feedback) using data we can observe (browsing behavior).


On Fri, Jan 22, 2016 at 11:19 AM, Trey Jones <> wrote:
Yesterday in the quarterly review Dan mentioned that our current user satisfaction metric uses the somewhat arbitrary 10s dwell time cutoff for a successful search, and that we want to use a survey to correlate qualitative and quantitative values to pin down a better cutoff for our users. I don't remember whether Dan mentioned it, or I was just rehashing the notion on my own, but it may be difficult to pin down a specific cutoff.

A wild thought appears! Why do we have to pin down a specific cut off? Why can't we have a probabilistic user satisfaction metric? (Other then complexity and computational speed, which may be relevant.)

We have the ability to gather so much data that we could easily compute something like this: 20% of users are satisfied when dwell time is <5s, 35% for 5-10s, 75% for 10-60s, 98% for 1m-5m, 85% for 5m-20m, and 80% for >20m.

Determining the cutoffs might be tricky, and computation is more complex than counting, but not ridiculously complicated, and potentially much more accurate for large samples. Presenting the results is still easy: "54.7% of our users are happy with their search results based on our dwell-time model".

I tried to do a quick search for papers on this topic, but I didn't find anything. I'm not familiar with the literature, so that may not mean much. 

Okay, back to the TextCat mines....


Trey Jones
Software Engineer, Discovery
Wikimedia Foundation

discovery mailing list

Mikhail Popov // Data Analyst, Discovery

Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment. Donate.