Using Virtual Machines is a too big overhead compared to just coding it right, and still it would not protect against eg. javascript injection.
Looking into LilyPond exception, I don't see any big problem: - It relies in Math variables for storing the files in the same folder (it was made before Math extension was split). - $wgMathPath isn't properly escaped, but that's minor. - Usage of hardcoded text, math_failure, <b>, etc. in error messages. - It uses escapeshellarg instead of wfEscapeShellArg but the filename is safe anyway (and our servers aren't windows). - Maybe of greater concern is that it assumes to own everything in $wgTmpDirectory when those files could have been created: a) By another extension b) By another instance of LilyPond
I don't know why it needs to trim the images generated by LilyPond, but there's probably a reason for that. Assuming that LilyPond code doesn't allow to open files, or execute programs, the current version of LilyPond is apparently safe.
Although I have to admit that it is not pretty, and its "store files without tracking" is something that we shouldn't repeat with new extensions.