I for one don't like plugins, but agree that they *can* be of *additional* use. So, why not display the molecule as a PNG, then open a new browser window with a 3D/whatever applet when the user clicks on it? alt="Click here to see it in 3D (EvilJava required)." would give a hint.
That might be the best compromise.
My opinion on applets for the wiki is they're fine so long as they run in the GNU RTE and the 'non-rich' content is still provided.
Going OT from SMILES, but for articles on, for example, AVL trees there are quite a few good, free applets that let the user add/remove nodes from an animated diagram. Being able to see the nodes move is a good visual aid, and makes it a lot easier to get a feel of the overall thing than a text + static image description.