Dear all,
Last week we ran an A/B test over changes to how our search system
provides results (and we look forward to sharing the results of that
with you shortly). Today, we're launching a second A/B test.
This test looks at the "phrase slop" setting within our search
infrastructure, a thoroughly disgusting term that simply refers to the
"distance" in words between a search query and a match. For example,
the search term "Ben Folds Five" has a "phrase slop" of 0 to the
match
"Ben Folds Five". The search term "Ben Five" has a "phrase
slop" of 1
- there is 1 word (Folds) in distance between the search term and the
match.
What we are going to do is experiment with altering the phrase slop
settings for 3% of users. 1% will have a slop of 0 (the current
default), 1% a slop of 1, and 1% a slop of 2. The hope is that by
tweaking this setting we can measurably reduce the number of search
queries that return 0 results by broadening the conditions under which
something is considered a match.
This search will run for a week, and kick off at 4pm PST. Once it's
completed we'll share the results publicly, as is the norm for our A/B
tests. Huge thanks go particularly to Trey Jones and David Causse for
their feedback on what settings we could alter, Mikhail Popov for his
work on experimental design, and Erik Bernhardson for both his
feedback and turning around necessary changes to the search
infrastructure on such short notice. For context, we began planning
this test last Thursday morning - a one-week turnaround on design and
implementation is phenomenal.
Thanks,
--
Oliver Keyes
Count Logula
Wikimedia Foundation