Dear all,
Last week we ran an A/B test over changes to how our search system provides results (and we look forward to sharing the results of that with you shortly). Today, we're launching a second A/B test.
This test looks at the "phrase slop" setting within our search infrastructure, a thoroughly disgusting term that simply refers to the "distance" in words between a search query and a match. For example, the search term "Ben Folds Five" has a "phrase slop" of 0 to the match "Ben Folds Five". The search term "Ben Five" has a "phrase slop" of 1 - there is 1 word (Folds) in distance between the search term and the match.
What we are going to do is experiment with altering the phrase slop settings for 3% of users. 1% will have a slop of 0 (the current default), 1% a slop of 1, and 1% a slop of 2. The hope is that by tweaking this setting we can measurably reduce the number of search queries that return 0 results by broadening the conditions under which something is considered a match.
This search will run for a week, and kick off at 4pm PST. Once it's completed we'll share the results publicly, as is the norm for our A/B tests. Huge thanks go particularly to Trey Jones and David Causse for their feedback on what settings we could alter, Mikhail Popov for his work on experimental design, and Erik Bernhardson for both his feedback and turning around necessary changes to the search infrastructure on such short notice. For context, we began planning this test last Thursday morning - a one-week turnaround on design and implementation is phenomenal.
Thanks,
thanks for the quick summary Oliver. Let's get these tests archived no the wiki so that at the end of the quarter we aren't having to crawl through mailing list archives
On Thu, Aug 20, 2015 at 9:07 AM, Oliver Keyes okeyes@wikimedia.org wrote:
Dear all,
Last week we ran an A/B test over changes to how our search system provides results (and we look forward to sharing the results of that with you shortly). Today, we're launching a second A/B test.
This test looks at the "phrase slop" setting within our search infrastructure, a thoroughly disgusting term that simply refers to the "distance" in words between a search query and a match. For example, the search term "Ben Folds Five" has a "phrase slop" of 0 to the match "Ben Folds Five". The search term "Ben Five" has a "phrase slop" of 1
- there is 1 word (Folds) in distance between the search term and the
match.
What we are going to do is experiment with altering the phrase slop settings for 3% of users. 1% will have a slop of 0 (the current default), 1% a slop of 1, and 1% a slop of 2. The hope is that by tweaking this setting we can measurably reduce the number of search queries that return 0 results by broadening the conditions under which something is considered a match.
This search will run for a week, and kick off at 4pm PST. Once it's completed we'll share the results publicly, as is the norm for our A/B tests. Huge thanks go particularly to Trey Jones and David Causse for their feedback on what settings we could alter, Mikhail Popov for his work on experimental design, and Erik Bernhardson for both his feedback and turning around necessary changes to the search infrastructure on such short notice. For context, we began planning this test last Thursday morning - a one-week turnaround on design and implementation is phenomenal.
Thanks,
-- Oliver Keyes Count Logula Wikimedia Foundation
Wikimedia-search mailing list Wikimedia-search@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimedia-search
Yep; they'll be documented on Meta in the same way all research projects are.
On 20 August 2015 at 13:06, Tomasz Finc tfinc@wikimedia.org wrote:
thanks for the quick summary Oliver. Let's get these tests archived no the wiki so that at the end of the quarter we aren't having to crawl through mailing list archives
On Thu, Aug 20, 2015 at 9:07 AM, Oliver Keyes okeyes@wikimedia.org wrote:
Dear all,
Last week we ran an A/B test over changes to how our search system provides results (and we look forward to sharing the results of that with you shortly). Today, we're launching a second A/B test.
This test looks at the "phrase slop" setting within our search infrastructure, a thoroughly disgusting term that simply refers to the "distance" in words between a search query and a match. For example, the search term "Ben Folds Five" has a "phrase slop" of 0 to the match "Ben Folds Five". The search term "Ben Five" has a "phrase slop" of 1
- there is 1 word (Folds) in distance between the search term and the
match.
What we are going to do is experiment with altering the phrase slop settings for 3% of users. 1% will have a slop of 0 (the current default), 1% a slop of 1, and 1% a slop of 2. The hope is that by tweaking this setting we can measurably reduce the number of search queries that return 0 results by broadening the conditions under which something is considered a match.
This search will run for a week, and kick off at 4pm PST. Once it's completed we'll share the results publicly, as is the norm for our A/B tests. Huge thanks go particularly to Trey Jones and David Causse for their feedback on what settings we could alter, Mikhail Popov for his work on experimental design, and Erik Bernhardson for both his feedback and turning around necessary changes to the search infrastructure on such short notice. For context, we began planning this test last Thursday morning - a one-week turnaround on design and implementation is phenomenal.
Thanks,
-- Oliver Keyes Count Logula Wikimedia Foundation
Wikimedia-search mailing list Wikimedia-search@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimedia-search
Wikimedia-search mailing list Wikimedia-search@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimedia-search