I've written up an account of how the current parser treats apostrophes here: http://www.mediawiki.org/wiki/Markup_spec/BNF/Inline_text#Determining_the_be...
All I've done is read the code of doAllQuotes() and translate it from a procedural style (first replace blah, then iterate through...) into a more declarative style (four apostrophes end up getting rendered as X if the following is the case...).
The most interesting case is this one:
Take ''''four''' apostrophes and then throw '''''five unclosed apostrophes at them.
Normally, four apostrophes is treated as apostrophe followed by bold. But when the parser finds unbalanced bold *and* italics on the line, it goes looking for a bold to split. The first bold, which is now preceded by an apostrophe, is seen as a good candidate because it seems to be a single letter followed by a bold (as in the l'''idee'' case). So that bold gets split *again*. Meaning that the four apostrophes end up getting rendered as two apostrophes followed by italics.
I suspect this was not planned behaviour.
Steve