On Wed, Jul 23, 2008 at 3:06 PM, WJhonson@aol.com wrote:
Transparency. There is no benefit and a great drawback to not indexing. Not indexing makes it appear we are hiding something. That belief is already very prevalent among our critics, we don't want to feed them a ton of raw steak.
It's hard to hide things when our own search engine can find them too. And even if our own search was restricted to not search though some things, AfD for example (though not doing so would probably make a lot of people mad, and rightfully so), then someone interested enough could write a spider that ignores robot tags and fetches everything, then grep it.
Or hell, just download a database dump.