A partial outcome of the research in natural language processing
in the last decade is the representation of language as numeric
vectors, called word embeddings. These are used in large language
models such as Bert, Elmo, and (Chat)GPT. A peculiar aspect of
these numeric vectors is that they cluster semantically, so that
words for similar concepts (dog, puppy, pet) group together even
though their spelling is very different. This can be used for
"semantic" search. If a search query (dog) is converted to a vector,
it can search terms found in documents (e.g. wiki articles) that
have similar vectors and find those of similar content even though
the text doesn't match.
https://en.wikipedia.org/wiki/Word_embedding
Here are just two of very many videos that explain the concept:
https://www.youtube.com/watch?v=xzHhZh7F25I
https://www.youtube.com/watch?v=MUve9LiEAeI
Is there any ongoing work at WMF or around the Mediawiki software
to apply this new technique to search in Wikipedia?
--
Lars Aronsson (lars(a)aronsson.se, user:LA2)
Linköping, Sweden