Hi,
The robots.txt file can now be customized for all wikis. There's a central robots.txt file which is common for all projects and which is maintained by the server administrators of Wikimedia.
Additionally, there's now the possibility to add custom entries on a per-wiki-base. Add lines to the Mediawiki:robots.txt page on your project and they will be sent as part of robots.txt, too.
Best regards,
JeLuF