Hey all,
Some announcements around the A/B tests we have run (and will be
running) on the Wikipedia portal.
A couple of weeks ago we launched an initial test to identify if a
more prominent search box would improve the rate at which users
clicked through from the portal to our various projects (at the
moment, only a third of users do so). This was our first A/B test as a
team, and so we expected various process flaws to show up when we
implemented it.
The test was essentially a wash due to one unfortunate and
unfortunately fatal flaw :(. The implementation of the logging did
pick up events from both test groups, so our initial lightweight
data-checking passed, but it did /not/ pick up *search* events
specifically - the one population we absolutely wanted to track. As a
result the test group did not contain the events we needed it to. This
was a failure of both the implementation and the QA process at the
analysis end; there is egg on everyone's face. In the next iteration
we will be conducting far deeper checks, both on the code and on the
output. We have already expanded our documentation around testing and
implementing A/B tests, at
https://meta.wikimedia.org/wiki/Discovery/Testing#Guidelines
To avoid running behind schedule, and to ensure we are producing the
most user-optimised version of the portal we can, we will be launching
a *multi-variable* A/B test - technically an A/B/C test - on 4
January. This will test the existing version of the portal, a version
with the more prominent search box, and a version with a more
prominent search box and friendly search results (containing pictures
and brief summaries, as they do on mobile), against each other.
This test should run for a week, and be non-invasive - only 0.03% of
users should even notice anything has happened. When it's done, we're
very hopeful that we'll have a clear winner and can make the
experience of using the portal better for the remaining 99%, too :).
For the Portal team,
--
Oliver Keyes
Count Logula
Wikimedia Foundation