I think that picking isolated websites (gmail or medium) isn't enough to get a sense of what the average user's expectation is. These two particular examples aren't necessarily the best for other reasons: Google products and Gmail in particular have always had very engineer-minded keyboard shortcuts because engineers rule the culture at Google. That's not necessarily the best thing for accessibility if you don't have that culture. As for Medium, it's too new to have proven itself as something with good accessibility. Maybe a lot of people are getting confused by medium's interface, we wouldn't know.

I suspect that the reason why this keep coming up is that we're overriding a key that normally provides core browser functionality. Also, anyone who happens to have the opposite mental model than the one we cater for, either through their experience of the web, of their use of tablets, etc. will have a very hard time training themselves to remember to "do it right" in Media Viewer. It goes against their intuition and even if we visually inform them that they're "doing it wrong", they have to learn to use the tool, which is bad. Something like Media Viewer should be intuitive for everyone. Even if we pick the option that works for the majority, there will still be a non-trivial amount of users for whom the up/down arrow behavior will feel counter-intuitive. And every time it doesn't do what they expect, they get a little more annoyed at the product.