This is
https://bugzilla.wikimedia.org/show_bug.cgi?id=8473
The obvious solution is to make $wgArticleRobotPolicies work as
advertized and not be overpowered by hardwired code.
Also having users maintain private copies of includes/* usually lasts
until the next upgrade only, when different staff don't know about
previous tweaks.
Yes the user could also choose to maintain a sitemap, but that is
beside the point.