I would like to include myself as someone who knows what they are talking about but it might be a stretch! Anyhow, we implemented Lucene on a wiki with approaching 350k pages. The performance and types of search available are impressive and we are using the existing search tables in MySQL to feed the Lucene index. We perofrm an incremental synchronization every 15 minutes and the indexing is fast with a couple of thousand documents being indexed per minute. It is unlikely we will have that many in a 15 minute period so all should be fine. Things like language stemming, fishing returns fish and fishes etc, multiple language support, prioritization based upon various factors (title, occurrences of words near each other) and a host of other features. Memory is indeed a consideration but even so we are able to run this on a 1GB dedicated server and still see search response times well under 200mS. We will be moving to a bigger server but not as a result of Lucene. If you look at Lucene then also look to implement Solr which provides added functionality as well as the standard highlight search term within the teaser results and transaction management amongst multiple index servers. If you want to talk to the best in the industry go to sematext (www.sematext.com) which is run by Otis, one of the original participants in Lucene. We are VERY happy with our move to Lucene and will be adding Solr in the next couple of weeks. Hope some of this helps. Regards, Paul On 12/5/07 3:35 PM, "Jim Hu" <jimhu(a)tamu.edu> wrote:

Jim Hu wrote: Regarding indexes, Sphinx can be set up to use either a MySQL backend or it's own data format, which is the standard mode. It might be though that the SphinxSearch extension ( http://www.mediawiki.org/wiki/Extension:SphinxSearch ) uses the wiki's database to get the article extracts for the search page, since these extracts are not in the indexes. I have, btw, been impressed by Sphinx's indexing speed (something like 1000 pages / 6 sec) , as well as it's set of features and config options (mutli-language stemming et.c.), and think it looks very promising. Regards Samuel

All I know is the author of http://meta.wikimedia.org/wiki/Fulltext_search_engines#Ksana_Search_For_Wik… is a nice guy.

On 06/12/2007, Gabriel Millerd <gmillerd(a)gmail.com> wrote: Since the thread came up, I would like to ask if any of the current search engines will fix my current search problem; I have a lot of transclusion in my wiki, whereby the bulk of many 'resource' pages is dropped in via a specific 'data' templates. For example; A 'resource' page, http://biodatabase.org/index.php/Ensembl The corresponding 'data' template, http://biodatabase.org/index.php/Template:NARDatabase:Ensembl Its somewhat messy, but it is a requirement because of the copyright status of the source 'data'. So far so good, however, my searches turn up hits to the underlying data template, and not the corresponding resource page. For example; http://biodatabase.org/index.php/Special:Search?search=Wellcome+Trust&f… I thought about hacking the underlying index tables to redirect terms from the data templates to the resource pages, but I am not sure if that is the best answer. If I go with Lucene can I fix this problem, or will it require more hacking? Thanks for any help, and sorry for the long winded description, Dan. -- You too can join the BIOINFORMATICS fun! irc://freenode.net/#bioinformatics or via the web portal; http://www.acm.jhu.edu/cgi-irc/irc.cgi?chan=%23bioinformatics

On 06/12/2007, Gabriel Millerd <gmillerd(a)gmail.com> wrote: Yes, Google 'sees' the transcluded data on the resource page, but the internal MW search indexes do not. I already set the value of that variable to ensure that all searches are in all namespaces by default. Personally I don't see the point of namespace specific searches. For the vast majority of wikies this adds confusion (and nothing else). Cheers, Dan. -- You too can join the BIOINFORMATICS fun! irc://freenode.net/#bioinformatics or via the web portal; http://www.acm.jhu.edu/cgi-irc/irc.cgi?chan=%23bioinformatics

On Dec 6, 2007 4:28 PM, Emufarmers Sangly <emufarmers(a)gmail.com> wrote: I should do this, your right. I don't work there anymore in the same capacity actually and rewriting all this for FOSS would be a good project to do now that I have the 'free time'. Lot of things I didn't like in that system. But this is a closer rundown of what I wrote: SpecialScan * Created a Special page that handles the 'front end' functionality akin to the mnoGo scripts * Implemented functionality in the above to handle the new page creation workflow (asdf is not found, create? * This is click tracked so we know when people use it and what they searched for (like google basically) * Altered the Monobook to use the above over the original. * I think initially mod_rewrite was used as a cheap hack as well. Backend: * Use perl and tt to create a indexer file on demand and launch the indexer. taking a peak at the mediawiki database to grab namespaces and other hints quick. * the indexer does its normal multiple burns through of the wiki under the two major user ids public and private. the private one is limited to just single namespace in this case though so its not a full run. * in mnoGo you can make 'datasets' of various regex url patterns to search, so adjusting the result weighting or the revisit time of pages like ^/w/Talk: can be useful if your into that sort of thing. * special care to follow interwiki links to a limited degree and peek into the mediawiki data for 'okaying' them. SpecialScan/Terms * Special page for admins, since quieries are farmable making a special page for 'what did people most search for' and some cheap guesses at common failed searches goes here. SpecialMnogo/Config * Special page for admin people to tune the indexing process to some degree, purge them all, manually kick it off etc ... so I dont have to be involved. * Also handle the external urls here (mailing lists, and interwiki limiters) SpecialMnogo/Logs * Special page for admin people to see the indexing logs for what its worth -- Gabriel Millerd