Lars Aronsson wrote: I concur. In my own personal experience writing the search engine for Bomis and the old fastcgi Wikipedia search engine, it is not particularly costly to have on-the-fly scoring logic that's suitable for the problem at hand. Even some very ad hoc measures (scoring titles higher than the body is very powerful for Wikipedia, for example... at Bomis, I have a measure of "uselessness" for words that does a nice job of helping relevancy over a raw keyword search) can be hugely beneficial in having the search results cause joy in the searcher. I personally wonder about the raw performance of MySQL as compared to btree dbm files. I could, in theory, write a perl script to go through and index the current wikipedia database once per night, using some of my "tricks of the trade", and make the search engine a lot better than it is now. However, I wonder if that's the right approach. The downside to doing it my old-fashioned way is that the search engine only updates as often as I set the cron to update the search engine index. If the data is always "live" in the mysql database, and if it's just as fast, then that's obviously the better way to do it. I don't have any actual knowledge, I just know that fascgi perl + DB_File btree stuff is very fast. (Bomis handles about 100x the traffic on 3x the servers!) --Jimbo

Brion L. VIBBER wrote: A useful and simple (though not perfect) measure of uselessness is how many pages are returned for a given word. In English, 'a', 'an' and 'the' will appear in nearly every article. In Japanese, 'wa' and other similar marker words will appear in nearly every article. The more articles that are returned for a given search term, the less informative it is. --Jimbo