[Wikipedia-l] Okhrana and Google

[Vicki Rosenzweig]

At 11:41 PM 3/20/03 -0800, Brion wrote:
>On Thu, 2003-03-20 at 22:52, Zoe wrote:
> >> sannse <sannse at delphiforums.com> wrote:
> >> I just did a search on Google for "Okhrana" and came up with
> >> www.wikipedia.org/w/wiki.phtml?title=Okhrana&action=edit as the 10th
> >> hit. But that's a link to an edit page
> >
> > I think the problem was that Google had cached our page, and I just
> > deleted it.  You therefore got sent to a nonexistent entity.
>
>That wouldn't have gone to an edit page, just to a blank page. The
>problem here is that an actual edit URL got into google at some point
>and is still coming up in results.
>
>Sannse, we *do* exclude edit pages from google's and other bots'
>spiders, doubly:
>
>* robots.txt excludes access to the /w/ subdirectory, and thus all
>   direct script actions (edits, histories, diffs, printable mode,
>   changing options/length on recentchanges, etc), so it shouldn't be
>   touching them at all.

Google is in the habit of ignoring robots.txt files. (This keeps coming
up on LiveJournal, where the support volunteers have to explain to people
that even if they've asked not to be indexed, they'll have to contact
Google and ask to be removed.)
-- 
Vicki Rosenzweig
vr at redbird.org
http://www.redbird.org