It just seems to me that if MWiki allows custom tags, then it should allow a reasonable number of embeddings of custom tags within custom tags.
The MWiki code doesn't know what to expect inside the custom tags (wikitext, html, plain text, some kind of delimited data, etc), only the extension code knows that. Therefore, shouldn't the extension be the one to handle parsing the contents of custom tags? Just pass $input through $parser before doing anything with it if you are expecting wikitext as input.