Hi,
The robots.txt file can now be customized for all wikis. There's a
central robots.txt file which is common for all projects and which is
maintained by the server administrators of Wikimedia.
Additionally, there's now the possibility to add custom entries on a
per-wiki-base. Add lines to the Mediawiki:robots.txt page on your
project and they will be sent as part of robots.txt, too.
Best regards,
JeLuF