Hey all,

Some announcements around the A/B tests we have run (and will be running) on the Wikipedia portal.

A couple of weeks ago we launched an initial test to identify if a more prominent search box would improve the rate at which users clicked through from the portal to our various projects (at the moment, only a third of users do so). This was our first A/B test as a team, and so we expected various process flaws to show up when we implemented it.

The test was essentially a wash due to one unfortunate and unfortunately fatal flaw :(. The implementation of the logging did pick up events from both test groups, so our initial lightweight data-checking passed, but it did /not/ pick up *search* events specifically - the one population we absolutely wanted to track. As a result the test group did not contain the events we needed it to. This was a failure of both the implementation and the QA process at the analysis end; there is egg on everyone's face. In the next iteration we will be conducting far deeper checks, both on the code and on the output. We have already expanded our documentation around testing and implementing A/B tests, at https://meta.wikimedia.org/wiki/Discovery/Testing#Guidelines

To avoid running behind schedule, and to ensure we are producing the most user-optimised version of the portal we can, we will be launching a *multi-variable* A/B test - technically an A/B/C test - on 4 January. This will test the existing version of the portal, a version with the more prominent search box, and a version with a more prominent search box and friendly search results (containing pictures and brief summaries, as they do on mobile), against each other.

This test should run for a week, and be non-invasive - only 0.03% of users should even notice anything has happened. When it's done, we're very hopeful that we'll have a clear winner and can make the experience of using the portal better for the remaining 99%, too :).

For the Portal team,