On Fri, Mar 14, 2014 at 5:23 PM, Federico Leva (Nemo) <nemowiki@gmail.com> wrote:

Fabrice Florin, 15/03/2014 00:13:

We would appreciate your advice on our upcoming research study of image
load times on Media Viewer. [...]
*I. Goals*
The goal of this study is to determine whether or not Media Viewer is
loading images fast enough for the majority of our users in most common
situations.

As a typical user of the Media Viewer, I want images to load quickly, in
just a few seconds, so I don't have to wait a long time to see them. [...]

Definitions: [...]
*II. Questions* [...]

*III. Outcomes* [...]

I'm confused. Too many questions, too many arbitrary definitions, axioms. No falsifiability. I understand the idea of defining a minimum quality standard to respect, it might even be the only way, but it's a thicket that moreover only indirectly verifies what we're actually interested in.
At its root is simple, we need to know if readers enjoy the images/media more or get annoyed and don't look at them because they're too slow. (Measuring the value they get from the media, or attach to the page in consequence hence becoming more likely to visit the project more, is less clearcut.) So maybe there is some simple check for this, if surveys don't work maybe just the total number of requests or something.

Nemo

(Cross-posting to multimedia list since I think my reply there was bounced.)

Hey Federico,

I hope you'll forgive me for some shameless (but brief) thread-jacking that won't answer any of your questions but instead raise a couple of tangential points. Apologies for the derail.

I regret not being subscribed to the multimedia list (a mistake I plan to rectify immediate after sending off this e-mail) and thus not having had the chance to respond sooner. I haven't yet had a chance to review the results, but I did have a chance to closely review the instrumentation code that the Multimedia team devised for collecting these measurements, and I can tell you that it is extremely sophisticated, making use of a web performance API, the specification of which has graduated to W3C recommendation status *last month*. I don't know yet if any errors were made in the statistical sampling and aggregation of data (and I want to be careful not to suggest that there have been any), but I do want to stress that this represents a major technical achievement and that I think we will glean a lot of insight about the performance of multimedia content on Wikimedia wikis from this infrastructure for years to come. So, kudos for that!

---

Ori Livneh

ori@wikimedia.org