A change was just now made to the shared proxy system for Toolforge
which makes the proxy respond with default content for /favicon.ico
and /robots.txt when a tool's webservice returns a 404 Not Found
response for these files.
The default /favicon.ico is the same as
<https://tools-static.wmflabs.org/toolforge/favicons/favicon.ico>.
The default robots.txt denies access to all compliant web crawlers. We
decided that this "fail closed" approach would be safer than a "fail
open" telling all crawlers to crawl all tools. Any tool that does wish
to be indexed by search engines and other crawlers can serve their own
/robots.txt content. Please see <https://www.robotstxt.org/> for more
information on /robots.txt in general.
These changes fix a regression [0] in functionality caused by the
toolforge.org migration and the introduction of the 2020 Kubernetes
ingress layer. Previously the /robots.txt and /favicon.ico from the
"admin" tool were served for all tools due to the use of a shared
hostname.
[0]:
https://phabricator.wikimedia.org/T251628
Bryan, on behalf of the Toolforge admin team
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce