Re: [Wikimedia-search] Announcing a second round of A/B tests

Dear all, Last week we ran an A/B test over changes to how our search system provides results (and we look forward to sharing the results of that with you shortly). Today, we're launching a second A/B test. This test looks at the "phrase slop" setting within our search infrastructure, a thoroughly disgusting term that simply refers to the "distance" in words between a search query and a match. For example, the search term "Ben Folds Five" has a "phrase slop" of 0 to the match "Ben Folds Five". The search term "Ben Five" has a "phrase slop" of 1 - there is 1 word (Folds) in distance between the search term and the match. What we are going to do is experiment with altering the phrase slop settings for 3% of users. 1% will have a slop of 0 (the current default), 1% a slop of 1, and 1% a slop of 2. The hope is that by tweaking this setting we can measurably reduce the number of search queries that return 0 results by broadening the conditions under which something is considered a match. This search will run for a week, and kick off at 4pm PST. Once it's completed we'll share the results publicly, as is the norm for our A/B tests. Huge thanks go particularly to Trey Jones and David Causse for their feedback on what settings we could alter, Mikhail Popov for his work on experimental design, and Erik Bernhardson for both his feedback and turning around necessary changes to the search infrastructure on such short notice. For context, we began planning this test last Thursday morning - a one-week turnaround on design and implementation is phenomenal. Thanks, -- Oliver Keyes Count Logula Wikimedia Foundation _______________________________________________ Wikimedia-search mailing list Wikimedia-search(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimedia-search