[Labs-l] Google bot

[Wiki Billinghurst]

For the majority of the tools, it is finding the tools in Google in
the first place, and their help pages secondly.  If we can cover those
aspects, and then shut them out of the rest as the default, that
sounds like a good plan.

Regards, Billinghurst

On Mon, Oct 20, 2014 at 10:33 PM, Magnus Manske
<magnusmanske at googlemail.com> wrote:
> So what I imagine happens:
> * Someone finds a tool that is useful to run e.g. a query on any category
> with a certain template.
> * This is added as an auto-filled link in the template, for common
> maintenance tasks.
> * Googlebot follows all of these pre-filled links from Wikipedia and/or
> mirrors.
> * This does not just return the empty tool form, but actually runs the tool.
> * Horrible things ensue on Tools Labs.
>
> AFAIK, all tools use a default .lighttp configuration by default. Is that
> replaced or extended by a local config file?
> If it's replaced, the default config could exclude Googlebot, and even a
> blank local config file would re-enable Googlebot again, for those who want
> it.
>
>
>
> On Mon, Oct 20, 2014 at 2:33 AM, Maximilian Doerr
> <maximilian.doerr at gmail.com> wrote:
>>
>> I disagree.  Google bot has been nothing but a nuisance, continuously
>> probing my tool with different queries.  It has drained resources
>> needlessly, and was quite glad that tool labs had it blocked.
>>
>> Cyberpower678
>> English Wikipedia Account Creation Team
>> Mailing List Moderator
>>
>>
>>
>> On Oct 19, 2014, at 21:30, Nuria Ruiz <nuria at wikimedia.org> wrote:
>>
>> Why would we want to restrict google indexing of the whole cluster? There
>> are tools of many different nature deployed there, seems that indexing or
>> not should be configured on a instance per instance basis.
>>
>>
>>
>> On Sun, Oct 19, 2014 at 4:41 PM, Maximilian Doerr
>> <maximilian.doerr at gmail.com> wrote:
>>>
>>> Who protested to that, and why would that be a problem?
>>>
>>> Cyberpower678
>>> English Wikipedia Account Creation Team
>>> Mailing List Moderator
>>>
>>> -----Original Message-----
>>> From: labs-l-bounces at lists.wikimedia.org
>>> [mailto:labs-l-bounces at lists.wikimedia.org] On Behalf Of Marc A. Pelletier
>>> Sent: Sunday, October 19, 2014 7:29 PM
>>> To: labs-l at lists.wikimedia.org
>>> Subject: Re: [Labs-l] Google bot
>>>
>>> On 10/19/2014 03:50 PM, Magnus Manske wrote:
>>> > I vaguely remember that indexing bots (like the Google one) were
>>> > filtered out by Labs already?
>>>
>>> They were, for some time, but then I got some fairly vehement
>>> protestations that tools being unindexed by Google was a problem.
>>>
>>> -- Marc
>>>
>>>
>>> _______________________________________________
>>> Labs-l mailing list
>>> Labs-l at lists.wikimedia.org
>>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>>
>>>
>>> _______________________________________________
>>> Labs-l mailing list
>>> Labs-l at lists.wikimedia.org
>>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>
>>
>> _______________________________________________
>> Labs-l mailing list
>> Labs-l at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>
>>
>>
>> _______________________________________________
>> Labs-l mailing list
>> Labs-l at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>
>
>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
>