This is https://bugzilla.wikimedia.org/show_bug.cgi?id=8473
The obvious solution is to make $wgArticleRobotPolicies work as advertized and not be overpowered by hardwired code.
Also having users maintain private copies of includes/* usually lasts until the next upgrade only, when different staff don't know about previous tweaks.
Yes the user could also choose to maintain a sitemap, but that is beside the point.