Parsing of the pathological cases doesn't seem specifiable, but a simplified version probably would be.
What if we only allowed ''italic'', '''bold''' and ''''bold italic'''', and required a separator between consecutive markup. I.e. ''a''<s/>'''b''' => <i>a</i><b>b</b>; ''a'''''b''' => <i>a'''</a>b<b>..?
What if we didn't allow nesting, so ''italic and '''bold''''' had to be written as ''italic and ''<s/>''''bold''''?
That would probably go along way toward making it specifiable, without affecting 99% of the current text.
I think it's been agreed that outright rejecting any wikitext is a bad idea. Error messages or not, the parser has to at least try.