I'm only a week late to the party—and it's Friday the 13th so anything goes, right?
Erik wrote:
I would prefer to see us focus on search relevance and improving the scoring of what we already have before spending more focus on interwiki search.
David wrote:
Working without a relevancy lab will always lead to discrepancies like that, the developer will focus on a limited set of 4/5 queries to develop the feature with a high risk to break previous features. I'd really like to use the relevancy lab to review existing features.
David's uncovered a number of weird results with the standard search config (as have others), and while I love to say, "the plural of anecdote is not data", but that's David's point, we need to assess performance overall, not just on the motivational examples.
The relevance lab will let us test a lot of options quickly and relatively cheaply. I've been thinking about it in my 10% time and I've got a line on how to handle annotations (including "required result") that even in the absence of a proper gold standard corpus makes it feasible to collect examples like David's and use them as "search quality unit tests" to make sure we don't break things.
The cross-language cross-wiki task is endlessly fascinating (to a language nerd like me), but I worry that the maximum potential impact is low, and that success is very hard to measure, because the plausible use cases are so complex (esp. right now—I want inline surveys, dagnabbit!). I think the same may be true of other cross-wiki searching.
I think this also fits with the overall Discovery vision of first looking inward and making sure our fundamentals are sound.
Can we talk more about the theory and practice of updating our Q3 goals in our next weekly meeting?
—Trey