-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Mohamed Magdy wrote:
Robert Stojnic wrote:
Sounds nice to limit the search in certain category.. nice work! but what does this mean? "and stemmed words are penalized."
The stemming issue is reported in bug 2511 [*]. The bug is caused by the indexer not indexing the original word, but only it's root (i.e. stemmed word). Now both are indexed, and original words are preferred, i.e. have larger scores.
I may be wrong .. but isn't it right that before the program could get the root of the word it have to know it? i mean.. it should have a big list of words and its roots? and that is not for english only..you have to have lists for each language? or where else the program will strip the words
Roughly speaking, stemming is the process of taking inflected forms of words ("category" -> "categories") and extracting a normalized root form (say, "categori") for comparison purposes. This allows you to search on one form and receive results containing the other.
The exact code to do this will vary depending on language. A number of preexisting stemming filters exist for Lucene's indexing engine, some of which are used here.
Our currently-live search does basic stemming for English, German, Russian, and Esperanto, but not for other languages.
The issue Robert mentioned was that the old way would return results with any inflected form unconditionally, which can be annoying when you really did want an exact match. The new code has a preference for exact matches in the ranking, but will return related inflected forms as well.
- -- brion vibber (brion @ wikimedia.org)