On Tue, May 21, 2002 at 10:38:18AM -0700, Brion L. VIBBER wrote: It doesn't (unless you have changed this in my code, I didn't check). The parsing function usses the PERL regular expression \w to decide if something is legal in a search word or not. If it thinks a character is illegal it gives an error so you should have noticed that. If the result is empty and you didn't get a syntax error there can be two problems: - MySQL doesn't index that character. I wasn't able to find out which characters are exactly indexed by the full-text index and which not. I know that characters with umlauts get indexed (search for "G"odel" for example) but I wouldn't know about the ringel-S (sp?) for example. The easiest way to find out is probably simply trying. - The special characters in the articles were written by using entities. The search doesn't know that "o and ö are the same. Btw, if you have questions about the parse function for the search, just ask. I'm very busy at the moment so I don't have time to do any real programming, but answering a few questions should not be a problem. -- Jan Hidders

On mer, 2002-05-22 at 10:36, Kurt Jansson wrote: Nicht so gut, but I'll try to muddle through the list mit mein Wörterbuch und verstehen ein bisse... If I actually make the software check return codes and report useful error messages, I see: You have an error in your SQL syntax near 'MATCH (cur_ind_title) AGAINST ("Fläche") ) AND cur_t' at line 3 Looks like mismatched parentheses. I've replaced (what I think is) the rest of the \w regexps with [\w\x80-\xff], and it looks improved, but as much of the search code is a mystery to me, I can't guarantee anything! (Some changes to special_dosearch.php now in CVS.) (Whew!) Oh, that's so much nicer than "it's too slow!" or "that's a well-meaning feature but poorly implemented" or "make sure everything works 100% before putting it online". You're my new favorite user! ;) -- brion vibber (brion @ pobox.com)

On mer, 2002-05-22 at 14:43, Jan.Hidders wrote: If you'll recall, we're switching all the wikipedias to UTF-8 (if we don't do it now, we'll just end up doing it in a few years and it'll be more painful). The above covers some, but not all UTF-8 sequences that encode valid letters. In an ideal world, locale settings would apply (and work correctly and consistently!) and \w would match everything necessary, but... (PHP 4.1.0 has some sort of special UTF-8 mode for regexps that might or might not be useful here.) Oh, I don't doubt that at all. :) -- brion vibber (brion @ pobox.com)

On ĵaŭ, 2002-05-23 at 02:21, Jan.Hidders wrote: Fulltext search is already broken in a million ways! It doesn't know character references (ü, ĉ, į etc), it can't find partial matches or sounds-likes, it can't find "X" when you search for "Xs" or "Xs" when you search for "X", it doesn't return *ANY* results for words it thinks are too common... UTF-8 is the least of our problems; it just means that case-folding is a little trickier (and if we had a decent $*#@%# database, it would take care of that for us). Yes, that's *exactly* why we have to break out of the "everything in the world should be Latin-1, oh and by the way even though we have limited support for other character sets -- but not the ones YOU need -- you can only select one single character set for the whole database server! MWOOHAAHAAHAAA!" rut and make it work for the rest of us too. The particular problem I was discussing was the regexps, which were a PHP problem. MySQL could do all the magic it wanted; if the words don't get through the regexps in the PHP code they'll never get anywhere in the database's fulltext search. Oh, it's not like that hasn't been suggested. If anybody knows how to go about switching to Postgres, I sure as heck wouldn't object. I have no emotional attachment to MySQL; as far as I know it's only being used because Magnus was already familiar with it. -- brion vibber (brion @ pobox.com)

On ĵaŭ, 2002-05-23 at 04:52, Jan.Hidders wrote: What's not well-defined about "We're having some problems with the search engine right now; if you don't find what you're looking for at first, try capitalizing it and search again." ? If it is in fact having problems indexing chars over 128, install the attached hacked character set definition file and reindex the database. my.cnf -> /etc/ (or wherever) Index, custom.conf -> /usr/share/mysql/charsets/ (or wherever) Run 'myisamchk -r -q' over the tables. Works great on my machine... -- brion vibber (brion @ pobox.com)

On Thu, May 23, 2002 at 09:33:21PM +0200, Jan.Hidders wrote: An interesting alternative might be to do the indexing ourselves. This may sound very inefficient but let me explain. One method/trick that was used in PgSQL to get a fulltext index was to create someting like a separate table text_index( search_word, document_id ) with a conventional index on the column search_word. Now, every time a row is added to the table that you want to be searchable you retrieve all the search words (and their suffixes) and add the corresponding pairs to the table text_index. So if the text in document with id '5' is "Text Indexing" you add the following pairs to table text_index: ( "text", 5 ) ( "ext", 5 ) ( "xt", 5 ) ( "indexing", 5 ) ( "ndexing", 5 ) ... If you now want to search for an article containing "dex" you can query the text_index table and use in your WHERE clause something like "search_word LIKE 'dex%'" In this case the index for the search_word column is used because the LIKE expression doesn't start with a wild-card. As you can see this will also give you the partial matches. In PgSQL this is easy to implement because you can define triggers that call certain functions whenever a row is inserted or deleted in a table. But also in MySQL we could do this, although we would then have to ensure ourselves that every time an article is created/updated/deleted the text_index table is also updated. The nice part is that we would have total control over what is indexed and how it is indexed in the form of a PHP function that determines which pairs are added to text_index. This function could take care of collation and could even do nifty things like taking entity references into account or let a-umlaut match with ae. The different Wikipedias would probably all have a localized version of this function. The down-side is of course some performance when articles are updates and the extra work we need to do on writing this PHP function (although initial version could be quite simple). -- Jan Hidders

The current Nupedia software seems pretty unusable at the moment (copyeditor pages, among others). Personally, I doubt it will see the rebirth of Nupedia. As I keep advertising ;) I'm rewriting the whole thing (http://nunupedia.sourceforge.net). I doubt any MySQL-specific queries will be necessary in that project, so there's no need to keep MySQL just for Nupedia's sake. As for Wikipedia, except for the search function, I don't think there's much (if any) MySQL-specific stuff in there. Shouldn't be too hard to change to something else. Maybe we could use the Lee's test server for trying??? Magnus

On 5/23/02 1:34 PM, "Jimmy Wales" <jwales(a)bomis.com> wrote: I gotta say, I'm hardly convinced that Postgre is that astoundingly better than MySQL for our purposes. As was pointed out, Slashdot runs on MySQL, and it's a damn fast, high-traffic, complicated site. If you guys want to change to Postgre, go ahead, but it should be recognized that it's a mostly arbitrary decision. At least that's how I see it. --tc

On mar, 2002-05-21 at 06:28, Kurt Jansson wrote: Okay, I finally got a chance to finish that up. I've just checked in recaseLinks.php, which looks through the link tables with the critical eye of the old capitalization function, counts up the various alternate name forms in use for each page, and renames articles to the form most frequently used in links. It also puts in redirects for other forms that were used, so there shouldn't be breakage. Jason, if you could give this a try on the German test site: php recaseLinks.php php rebuildLinks.php # The link tables then need to be rebuilt (The Esperanto wiki also needs it, but it still needs to be converted again to see if the alternate charset function works as I think it should. I'd be mucho grateful if you could do that, so I can set people loose on bug & message-translation checking!) -- brion vibber (brion @ pobox.com)