This has been done before, for instance in the ASCIIMath4Wiki extension [2]. I don't want to change the Content-type unconditionally, though, only some of the time, so that we can serve texvc-style images to browsers or users that don't like the modified content type.
Note that this will interfere with any kind of HTML caching, such as Squid or file cache, and with the parser cache as well. It won't work correctly in most well-configured MediaWiki installs unless you make sure to fragment the parser cache appropriately, at a minimum.
The patch seems to work with the simple caching on the system I'm using to test. It changes more than just the content-type - the DOCTYPE is different and there's a <?xml string before the document, and a 'Vary: Accept' header.
I got all that from another author. I'll look further into how to do it without patches. It's true I can change the Content-type by setting $wgMimeTypes dynamically, but it only works on the first hit, and cached pages arrive as text/html.