Steve Bennett wrote: Currently it works off the page data dumps. -- brion vibber (brion @ pobox.com)

On 13/04/06, Jakob Voss <jakob.voss(a)nichtich.de> wrote: That would be more than fine. I gather the search db is currently several months out of date? But that wasn't my major complaint. Well, it should be as easily accessible as the search box is now. That would be nice, but even the simple mechanism of exact matches would be a start. And then you can add fall backs, like all upper case, all lower case, upper case first letter of each word and so on. If performance is the issue here. Steve

Jakob Voss wrote: I'm not sure if you're talking about the big web search engines (Google, Yahoo, MSN) or the search function in Mediawiki here. There is little excuse for the latter to have any delay. But even for a big web search engine, it is easy to keep track of how often each webpage has changed in history, and economize how often it needs to be revisited. Combined with the high PageRank of en.wp's RecentChanges (9 of 10), it would be trivial for Googlebot to revisit this page (or the front page of websites of major newspapers) every minute or two and make it a high priority to reindex all pages linked from there. I suppose this is how Google News works. Why it still takes about a month for Google to update its index on Wikipedia articles is a mystery to me. Probably it has to do with a lack of competition. If MSN or Yahoo were faster, it would force Google to improve. -- Lars Aronsson (lars(a)aronsson.se) Aronsson Datateknik - http://aronsson.se

On April 14, Evan Martin wrote: As far as I understand the robots meta tag, "noindex,follow" tells robots that they are welcome to fetch the page, that they can find links to other pages here (= follow), but they should never show this page among the search hits (= noindex). Words such as crawl and index are somewhat fuzzy here. Does "index" mean fetch or does it mean store in an index, to be returned to users as a search hit? I found no clear answer. Of course, the crawler/robot/spider is already fetching the page when it sees the meta tag. And it must fetch the page again to see if the meta tag has changed. The Pipermail software that is used for the wikitech-l archive sets "noindex,follow" for the overview sorted by date, e.g. http://mail.wikimedia.org/pipermail/wikitech-l/2006-April/date.html but for the individual posting, it sets "index,nofollow", e.g. http://mail.wikimedia.org/pipermail/wikitech-l/2006-April/034969.html I believe that "noindex,follow" is used for many "sitemap" pages, and this is my idea of how search robots should use RecentChanges. Indeed, the front page of any newspaper website is also similar to a sitemap. Its content changes so often that it becomes useless to index it under any specific word found there. If people search for "hurricane katrina", they don't want the front page of the Washington Post, which will have changed by the time they arrive. But they might be interested in the news article about this topic, and the front page was the way to harvest the link to that article. The main difference, then, between the newspaper and Wikipedia is that the newspaper uses their RecentChanges as their front page. Plus the fact that Wikipedia isn't covered by Google News. -- Lars Aronsson (lars(a)aronsson.se) Aronsson Datateknik - http://aronsson.se