At 11:41 PM 3/20/03 -0800, Brion wrote:
On Thu, 2003-03-20 at 22:52, Zoe wrote:
sannse sannse@delphiforums.com wrote: I just did a search on Google for "Okhrana" and came up with www.wikipedia.org/w/wiki.phtml?title=Okhrana&action=edit as the 10th hit. But that's a link to an edit page
I think the problem was that Google had cached our page, and I just deleted it. You therefore got sent to a nonexistent entity.
That wouldn't have gone to an edit page, just to a blank page. The problem here is that an actual edit URL got into google at some point and is still coming up in results.
Sannse, we *do* exclude edit pages from google's and other bots' spiders, doubly:
- robots.txt excludes access to the /w/ subdirectory, and thus all direct script actions (edits, histories, diffs, printable mode, changing options/length on recentchanges, etc), so it shouldn't be touching them at all.
Google is in the habit of ignoring robots.txt files. (This keeps coming up on LiveJournal, where the support volunteers have to explain to people that even if they've asked not to be indexed, they'll have to contact Google and ask to be removed.)