A change was just now made to the shared proxy system for Toolforge which makes the proxy respond with default content for /favicon.ico and /robots.txt when a tool's webservice returns a 404 Not Found response for these files.

The default /favicon.ico is the same as https://tools-static.wmflabs.org/toolforge/favicons/favicon.ico.

The default robots.txt denies access to all compliant web crawlers. We decided that this "fail closed" approach would be safer than a "fail open" telling all crawlers to crawl all tools. Any tool that does wish to be indexed by search engines and other crawlers can serve their own /robots.txt content. Please see https://www.robotstxt.org/ for more information on /robots.txt in general.

These changes fix a regression [0] in functionality caused by the toolforge.org migration and the introduction of the 2020 Kubernetes ingress layer. Previously the /robots.txt and /favicon.ico from the "admin" tool were served for all tools due to the use of a shared hostname.

[0]: https://phabricator.wikimedia.org/T251628

Bryan, on behalf of the Toolforge admin team