The problem with french is slightly different but leads to somewhat the same problems: there are no ascii folding configured currently.
This leads to the same intitle problems where it can't find words with diacritics when diacritics are omitted in the query.
For french putting ascii folding before the stemmer is certainly
a bad idea and we should imo do ascii folding after the stemmer
(possibly using preserve original).
I'm less sure that re-ordering would do the right thing for French. Presumably the French stemmer knows about accented characters and uses them. We should test and make sure. Maybe we need custom folding only for "unusual" accents in any given language (they are all unusual for English).
We can test French the same way we tested English, though, and be sure.
—Trey
Trey JonesSoftware Engineer, Discovery
Wikimedia Foundation
On Wed, Aug 10, 2016 at 10:48 AM, David Causse <dcausse@wikimedia.org> wrote:
______________________________Thanks Trey!
this will certainly greatly improve the intitle keyword as it uses the field with stems for filtering and hopefully will find pages that were ignored because of this filter ordering (e.g. intitle:louys can't find User:Louÿs currently).
I think I'll do the same for French which suffers from the same problem. IMO we should continue to work on this for other languages while we try to switch from asciifolding (latin letters only) to icu folding.
We may require some guidance on some languages where diacritics removal can be counter productive and maybe blacklist some letters (e.g. for finnish: is it appropriate to fold Ä or Ö for example?)
Note on accent folding: cirrus tries to always prefer exact matches. Searching for élément should always prefer élément over element. Users that prefer exact matches can always force cirrus to discard stems by wrapping the word in double quotes, e.g. "élément".
Le 10/08/2016 à 16:07, Trey Jones a écrit :
David and I had a discussion about moving ascii-folding to come before stemming on English Wikipedia. It seemed like a good idea, but we decided we should run some tests before implementing it, just to be sure.
Turns out it is a good idea!
Much more detail:
We won't deploy it until we deploy BM25 later in the year, since it requires a full re-index of English Wikipedia, as does BM25. That's something we should only do once.
—Trey
Trey JonesSoftware Engineer, Discovery
Wikimedia Foundation
_______________________________________________ discovery mailing list discovery@lists.wikimedia.org https://lists.wikimedia.org/ mailman/listinfo/discovery _________________ discovery mailing list discovery@lists.wikimedia.org https://lists.wikimedia.org/ mailman/listinfo/discovery _______________________________________________ discovery mailing list discovery@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/discovery