In today's Language Search sprint planning meeting, we came up with the following short-term plan (spanning roughly the next few days to a week):

The developers are making good progress implementing Treys "Minimum Viable Product" Relevance Lab[1], and if things go well, it should be functional soon. It will allow us to feed sets of queries in, have them run through two different search rules, and compare the results.

For now, those result comparisons will be able to objectively note Zero Results Rates, and the rest of the comparison will be subjective human "Were results A better than results B?" Such are the limitations of an MVP.

We have brainstormed a list of over a dozen possible changes that are candidates for our next A/B tests to try to improve language searching[2]. Erik is going to coordinate rating each of them for suitability for validation via the MVP relevance lab. Basically, for each one, whether it could be tested in the relevance lab, and if so, how much or little effort would be required to get to that point.

Once we know which ideas can be easily tested by the relevance lab, we should be able to run the best candidates through the lab, and use those results to decide which one(s) to take forward and implement as live A/B tests in production.

That's the strawdog plan, anyway.

[1] https://phabricator.wikimedia.org/T116872
[2] https://etherpad.wikimedia.org/p/LanguageSupportBrainstorming


Kevin Smith
Agile Coach, Wikimedia Foundation