Thanks Trey!
this will certainly greatly improve the intitle keyword as it uses the field with stems for filtering and hopefully will find pages that were ignored because of this filter ordering (e.g. intitle:louys can't find User:Louÿs currently).
I think I'll do the same for French which suffers from the same problem. IMO we should continue to work on this for other languages while we try to switch from asciifolding (latin letters only) to icu folding.
We may require some guidance on some languages where diacritics removal can be counter productive and maybe blacklist some letters (e.g. for finnish: is it appropriate to fold Ä or Ö for example?)
Note on accent folding: cirrus tries to always prefer exact
matches. Searching for élément should always prefer élément over
element. Users that prefer exact matches can always force cirrus
to discard stems by wrapping the word in double quotes, e.g.
"élément".
David and I had a discussion about moving ascii-folding to come before stemming on English Wikipedia. It seemed like a good idea, but we decided we should run some tests before implementing it, just to be sure.
Turns out it is a good idea!
Much more detail:
We won't deploy it until we deploy BM25 later in the year, since it requires a full re-index of English Wikipedia, as does BM25. That's something we should only do once.
—Trey
Trey JonesSoftware Engineer, Discovery
Wikimedia Foundation
_______________________________________________ discovery mailing list discovery@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/discovery