Steve Bennett wrote: Currently the index does not update on a regular basis as it caused the search servers to explode violently, bringing everything down. Until that is corrected, search is only updated sporadically. Click the "go" button or hit "enter" when searching. -- brion vibber (brion @ pobox.com)

Steve Bennett wrote: Currently it works off the page data dumps. -- brion vibber (brion @ pobox.com)

On 4/13/06, Jakob Voss <jakob.voss(a)nichtich.de> wrote: [snip] narf. Seqscanning through a million rows on each users query? I think not. It MySQL's full text indexing worked on anything but myisam tables it would be easy to provide your title search, but they don't.

On 4/13/06, Steve Bennett <stevage(a)gmail.com> wrote: All upper/lower isn't really effective on Wikipedia because most of our multiword titles are mixed case. AFAIR, most string matches in mysql are case insensitive, which would mean that we could have indexed case insensitive matches quickly... but I'm guessing that our use of binary fields for titles (which is required because no version of mysql has complete UTF-8 support) most likely breaks that. Alas, mysql doesn't have functional indexes (http://bugs.mysql.com/bug.php?id=4990) or it would be fairly trivial to offer a fast case-insensitive match.

On 4/14/06, Lars Aronsson &lt;lars(a)aronsson.se&gt; wrote: I know this was intentionally provocative, but I'll bite anyway. As far as I know, the limitation in general on Google indexing more of wikipedia is that wikipedia can't serve pages fast enough (or, more accurately, the extra load of more Googlebot will make wikipedia slower). To answer your specific proposal: 1) http://en.wikipedia.org/wiki/Special:Recentchanges has a meta tag: <meta name="robots" content="noindex,follow" /> which indicates it's explicitly disallowed from being crawled. 2) If it were allowed to be crawled, I'd expect it to be regularly updated for the reasons you describe. But even in that case, this particular page changing rapidly is not an indicator that the target pages are also changing rapidly. For example, I imagine that the digg front page changes pretty much every time a crawler visits, but the pages linked *from* digg are not necessarily changing any more rapidly than any other random page on the web is changing. Instead, there is a way for webmasters and Google to cooperate: the sitemaps program. You can read more about it here: https://www.google.com/webmasters/sitemaps/docs/en/about.html

On April 14, Evan Martin wrote: As far as I understand the robots meta tag, "noindex,follow" tells robots that they are welcome to fetch the page, that they can find links to other pages here (= follow), but they should never show this page among the search hits (= noindex). Words such as crawl and index are somewhat fuzzy here. Does "index" mean fetch or does it mean store in an index, to be returned to users as a search hit? I found no clear answer. Of course, the crawler/robot/spider is already fetching the page when it sees the meta tag. And it must fetch the page again to see if the meta tag has changed. The Pipermail software that is used for the wikitech-l archive sets "noindex,follow" for the overview sorted by date, e.g. http://mail.wikimedia.org/pipermail/wikitech-l/2006-April/date.html but for the individual posting, it sets "index,nofollow", e.g. http://mail.wikimedia.org/pipermail/wikitech-l/2006-April/034969.html I believe that "noindex,follow" is used for many "sitemap" pages, and this is my idea of how search robots should use RecentChanges. Indeed, the front page of any newspaper website is also similar to a sitemap. Its content changes so often that it becomes useless to index it under any specific word found there. If people search for "hurricane katrina", they don't want the front page of the Washington Post, which will have changed by the time they arrive. But they might be interested in the news article about this topic, and the front page was the way to harvest the link to that article. The main difference, then, between the newspaper and Wikipedia is that the newspaper uses their RecentChanges as their front page. Plus the fact that Wikipedia isn't covered by Google News. -- Lars Aronsson (lars(a)aronsson.se) Aronsson Datateknik - http://aronsson.se