I did some tests on
https://www.mediawiki.org/wiki/Lightbox_demo over a slowish mobile connection (1 Mbps downlink). Opening a lightbox takes over 30s with a warm JS cache when you do it immediately on page load (via an URL hashtag). The actual image load takes about 4s; the rest of the time is taken up by all the image thumbnails on the page loading. Since MediaViewer's bootstrap script runs on DOM ready, every single image on the page is queued to load before it; these are small ~200px thumbnails which take a few hundred milliseconds to load, but there are several dozens of them and they add up.
This is probably not what causes most delays, but it can cause delays for specific conditions (there are lots of thumbnails on the page, the user's browser did not cache them yet, and they use an URL hash or click on a thumbnail soon after the page loaded) which are not unplausible (e.g. a MediaViewer URL to an image on a gallery or category page is being shared on twitter). Unfortunately, I have no idea what to do with this one - the issue is not with anything MediaViewer does, and there is no way to influence the order in which the browser loads the images (we could try horrible hacks like loading the image in AJAX, but at best they would help on some browsers only). We could add more logic to the head script to start image preloading immediately, but eww. I suspect we will need a UI solution for this one, which tells the user to expect waiting for a while.