On Tuesday 29 March 2005 21:16, Minty wrote: there is a working prototype of a search engine using lucene (the framework nutch is built on). however, there are some political difficulties with using this - it's based on java, and java is not free. a number of people seem to value the fact that wikipedia runs completely on open source software higher than a working fulltext search. one _could_ try to port that code to mono and nlucene.. daniel -- I find the whole business of religion profoundly interesting. But it does mystify me that otherwise intelligent people take it seriously. - Douglas Adams

Minty wrote in gmane.science.linguistics.wikipedia.technical: look at the "lucene-search" module in CVS (http://cvs.sourceforge.net/viewcvs.py/wikipedia/lucene-search/). this is a (mostly) complete and functional Lucene (Java version) based search server for MediaWiki. i'm not sure how similar the Java version is to other versions, but you may be able to port the relevant bits without too much effort. an experimental test of this on the live site showed that it was able to handle our search load on a single 3.0GHz P4 with very minimal CPU usage, as long as the typo suggestion feature isn't enabled (because that uses several slow searches to produce the result; it could almost certainly be reimplemented in a much more efficient manner). kate.

On Tue, 29 Mar 2005 20:16:03 +0100, Minty <mintywalker(a)gmail.com> wrote: Actually, I think the more generic Lucene library which Nutch is built upon will be more useful. We should be indexing the wikitext, not the HTML (which is a lower quality version ;)) Seriously, we also don't want a crawler. What is left in Nutch's favour? However, I don't imagine either will be used by Wikimedia, as they are written in Java, which is slow and takes up too much memory compared to natively running stuff (i.e. C or C++). It's already bad enough that we're using PHP! (In one extreme case, a diff took 45.5 seconds in PHP while the same algorithm took 0.5 seconds in C (or maybe C++) (this is from a developer)).

This is the only open issue when you plan to use lucene, you need a good parser for the syntax and this is very difficult. Nothing! Use Lucene - trust me. :-) It will definitely save wikipedia very very much load!!!

Please perhaps you want to consider to visit the new page http://meta.wikipedia.org/wiki/FulltextSearchEngines when you want to participate the discussion about a new full text search engine ? The page and its talk page is awaiting your discussions. Tom

Thank you, Thomas, older versions of Joda are working since 1996 as news paper archive for the Rhein-Zeitung (Koblenz and Mainz, Germany). It's also used for archive and newsdesk purposes from several other european newspapers. At the moment it is going into action as full text index for europeans biggest magazine. It is also in use for the public index of the state archive of Rheinland-Pfalz (Germany). Last year I created two mirrors of WikiPedia, one using MediaWiki for demonstration purposes and another - our public one - using our own read-only web frontend. Joda is integrated into both mirrors: http://wikipedia.rhein-zeitung.de/index.php/Hauptseite (MediaWiki) http://lexikon.rhein-zeitung.de/ (our special Wikipedia interface) At the suggestion of Magnus Manske (not related :-) I published Joda under LGPL and made serveral improvements for the Wikipedia task. I wrote tools for indexing a whole cur table either from MySQL or from a SQL dump (which is twice faster). Indexing the german Wikipedia cur table (>210.000 articles, 36 million words) lasts approx. 45 minutes. An optional database optimization lasts additional 25 minutes. Both on a dual Athlon 2800+ machine with 1 GB RAM (the indexer is a multi threaded perl program). Joda can erase or update entries on the fly and can handle queries with parantheses and word distance operators like http://lexikon.rhein-zeitung.de/?((Albert OR Alfred) AND.1 Einstein) NEAR Quant*) NOT Gravitation. See more features under http://ioda.sourceforge.net/ Joda kernel is written with the Free Pascal compiler (http://sourceforge.net/projects/freepascal/). The tools are written in Perl. There a libraries for using joda directly from C, Perl, Python and PHP, all published under LGPL. The joda binaries are: command line program, TCP socket driven server and CGI. Yours jo

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Stefan Groschupf wrote: How does Google do it? :P Maybe we'll just have to deal with slow index times as a tradeoff to a fast, working (albeit slightly out of date) search. - -- Edward Z. Yang Personal: edwardzyang(a)thewritingpot.com SN:Ambush Commander Website: http://www.thewritingpot.com/ GPGKey:0x869C48DA http://www.thewritingpot.com/gpgpubkey.asc 3FA8 E9A9 7385 B691 A6FC B3CB A933 BE7D 869C 48DA

On Tue, 2005-03-29 at 13:02 -0800, Jimmy (Jimbo) Wales wrote: It should be pretty simple to use something like this. Now that we have several servers, we can dedicate one or more to text searches only (perhaps "search.wikimedia.org" on round-robin DNS or something) and not have any impact the wiki servers at all. I'll experiment a bit at let you know. -- Lee Daniel Crocker &lt;lee(a)piclab.com&gt; <http://www.piclab.com/lee/> <http://creativecommons.org/licenses/publicdomain/>

I don't know where things are/got-to wrt Google donating servers, but: Would they be able to donate one of their Search-Appliance thingies? Even if as a short term measure? + It'd allow for daily updates + It'd be no coding/dev work for WP + No extra software to install on your servers, no extra hardware to look after + Look n Feel could be customised, and I imagine it could even live under it's own wikipedia subdomain - Would involve Google crawling the site again - Might not deal well with older revisions / seperation of meta pages etc. If there is a philosophical objection to the "non free" java, I can at least see the potential for concern about doing this, but equally if we are getting hardware from them for free anyway it seems logically to get Search customised hardware+service, rather than just pure hosting hardware..... And in any event, there is no "lock in" if you ever wanted to revert to a different implementation.

Minty wrote in gmane.science.linguistics.wikipedia.technical: i've thought about this before. i didn't look at the appliances too closely, but my impression was that they could only index a relatively small number of documents (about 10,000). is this wrong? other than that, this would probably be the easiest way to solve the problem for us; OTOH it's not much use to other MediaWiki users. [...] the current situation appears to be that non-free software is not allowed, but software contained on other embedded devices is okay (e.g. switch firmware). given this i don't think there would be an issue with using one of the google devices. kate.