For search, there's a few different approaches for quantitative testing that are less difficult than A/B testing in terms of development overhead, data analysis and coordination. One of those is to replay real user queries against the index, but run the query with slightly different parameters from original. This is super cheap compared to an A/B test, but the downside is that it can only answer really deterministic (for lack of a better word) things, like how the parameters affect the zero-results rate or result ordering; since there's no user interaction with the replayed queries, you don't know what the clickthrough would've been, so it's hard to measure how satisfied the user would've been.

Hopefully that helps explain it.

Thanks,

Dan