[Foundation-l] [WikiEN-l] Searchability of non-mainspace pages

[Ryan]

On Tue, Apr 29, 2008 at 1:25 PM, Chad <innocentkiller at gmail.com> wrote:

> The way to do it would be to open a bugzilla request.
>
> -Chad
>

Right now, robots.txt excludes certain areas, including XfD, from external
search.  There are two problems with this:

1.  Internal search indexes them.
2.  Developers are forced to work on them on a case-by-case basis, which
takes a lot of time to work on.  For example, bug 13398, regarding
well-intentioned bot reports that tend to seem like nasty condemnations of
sites as "spammers", has been open for over a month, but there are many
other important bugs open, and so this and other robots.txt bugs have not
yet been fixed.

For the former problem, it seems fairly straightforward to have logged-out
users search only mainspace pages, either by default behavior of the search
box, or by banning them entirely from searching these pages.  Logged-in
users would retain their current defaults and ability to search user pages,
etc.

For the latter problem, I think the community needs to indicate that these
bugzilla requests are important, and should be handled much faster.  I have
concerns about blocking namespaces because of the possible affects on other
languages who'd rather not block these namespaces (because robots.txt works
globally), but frankly I think that a full block of other namespaces would
be a good idea, and should be considered on meta.

--
[[User:Ral315]]