Message mode (output type OT_MSG in the parser) was only meant to replace variables like {{SITENAME}} and parser functions like {{NS:4}}, it was never meant to allow templates. It's a very commonly called code path, and I wanted it to be efficient. Unfortunately, due to a bug apparently present since my initial version of the code, it did allow templates at the first level of recursion only. However, because various points in the code assumed that they weren't allowed, they were broken in various ways, such as the fact that triple-brace arguments aren't recognised and don't work.
In September 2004 in r5477, a second transformMsg() call was added to the output path of wfMsg(). transformMsg() is called once in MessageCache::getMessage(), and again in wfMsgGetKey() (originally wfMsgReal). So all messages containing double braces (except ones retrieved with wfMsgExt) are double-parsed.
Apparently nobody noticed this except for the users, who began to add a second level of template recursion to their messages, now enabled by the double-parse.
I had a half-hearted go at reproducing the oddities of message mode in the new preprocessor, but inevitably, there are some changes. If we're going to have changes, we may as well go the whole hog.
I propose removing message mode, which obviously has never been reviewed or tested properly, and replacing it with preprocess mode. So instead of calling transformMsg() twice, messages will be transformed by calling preprocess() once.
Preprocess mode at least gets some eyeballs on it due to its use in Special:ExpandTemplates, where there is a strict expectation of consistency with HTML mode. It works in a reasonably intuitive way: templates and template arguments work in basically the same way as in HTML mode, it's just XML-style extensions that get skipped.
-- Tim Starling