Hi everyone,
We would appreciate your advice on our upcoming research study of image load times on Media Viewer.
Here are proposed goals, questions and outcomes for this study. They are presented for discussion purposes, not as a prescriptive requirement - and will be adjusted based on your feedback.
I. Goals The goal of this study is to determine whether or not Media Viewer is loading images fast enough for the majority of our users in most common situations.
As a typical user of the Media Viewer, I want images to load quickly, in just a few seconds, so I don't have to wait a long time to see them.
Here are our recommended performance targets for image load times by connection speed, to match user expectations on the Web: * 1-2 seconds for a medium-size image on a fast connection * 2-3 seconds for the same image on a medium connection * 5-8 seconds for the same image a slow connection
If tracking connection speeds is too hard in our time-frame, we could base our performance targets on image size instead. For example: * 1-2 seconds for a small-size image on a medium connection * 2-3 seconds for medium-size image on the same connection * 5-8 seconds for large-size image on the same connection
Definitions: * Image load time = the number of seconds from when you click on a thumbnail to when you see the full image * Image size: large = over 2Mb, medium = 1 to 2Mb, small = under 1Mb * Connection speed: fast = over 256 Kbs, medium = 64 to 256 Kbs, slow = under 64 Kbs
The above numbers are for discussion purposes, and can be adjusted based on your feedback.
II. Questions Here are the main research questions we propose to answer about image load performance.
1. How long does it take for an image to load for the conditions below? (image load = total time from thumbnail click to full image display)
a. by image size: load times for large images? medium images? small images?
b. by web site: load times for mediawiki.org? commons? enwiki? frwiki? huwiki? other sites?
c. by connection speed: (optional) load times for fast connections? medium connections? small connections? (this may not be feasible in our time frame)
d. by daypart: (optional) load times for morning? afternoon? evening? night time? (to show if performance slows during peak hours)
This question could be answered by storing the timestamp for thumbnail clicks, as well as the timestamp for the full image display, then log the difference.
We would then prepare different bar graphs for each condition set above, with categories on the vertical axis, and number of seconds on the horizontal axis. The graphs could be based on data from the last 7 days.
2. How often does the image load time exceed our performance targets above?
a. by load time in a day: number of images that load in under 1 second? in 1-2 seconds? in 2-3 seconds? … and so on, up to 10 seconds or more
b. by load time in a week: number of images that load in under 1 second? in 1-2 seconds? in 2-3 seconds? … and so on, up to 10 seconds or more
This question could be answered by preparing different histograms, with number of images on the vertical axis, and number of seconds on the horizontal axis (deciles).
III. Outcomes To answer these questions, we plan to collect data during our upcoming pilots on different sites in April.
Based on these pilot results, we will need to make decisions about the wider deployments planned for May.
Here are possible outcomes from this study:
Outcome 1: Favorable - e.g.: 80% of images load quickly Action: Go ahead with current release plan to deploy Media Viewer everywhere by default.
Scenario 2: Neutral - e.g.: 50% of images load quickly Action: Go ahead with current release plan, but deploy Media Viewer as an opt-in feature on wikis that don’t want it by default
Scenario 3: Unfavorable - e.g.: 20% of images load quickly Action: Revisit release plan: consider making this opt-in everywhere — or work on faster image load solutions.
We would be grateful for your comments on this, so we can refine our plans before we start this study next week. Please let us know which metrics above seem most important, given that we only have a few developer days to collect and analyze a few key metrics in coming weeks, to determine if we are meeting our objectives. Some related links are included below, for your convenience.
To end on a positive note, we just deployed yesterday a new version of Media Viewer that is much faster, thanks to all the fine work from our development team. This morning, I looked at a variety on 'non-popular' images on enwiki today, and the Media Viewer experience was quite good overall. Most images load within the 2 second maximum which we recommend for a ‘fast’ connection — and this was a home wifi connection. I realize this is completely anecdotal, and not supported by hard data, so we can’t make any decisions about it. But it makes me hopeful that we are getting close to our objectives. Even compared to large commercial sites like Flickr, we hold up pretty well on this computer. :)
Thanks for your interest in this project.
All the best,
Fabrice
_______________________________
USEFUL LINKS
* Media Viewer Release Plan: https://www.mediawiki.org/wiki/Multimedia/Media_Viewer/Release_Plan
* First Media Viewer Metrics: http://multimedia-metrics.wmflabs.org/dashboards/mmv Metrics
* Media Viewer Test Page: https://commons.wikimedia.org/wiki/Commons:Lightbox_demo
* Metrics Tasks under consideration (Mingle): https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards?favorite...
* Next Development Cycle (Mingle): http://ur1.ca/gtvvr
* About Media Viewer: https://www.mediawiki.org/wiki/Multimedia/About_Media_Viewer
_______________________________
Fabrice Florin Product Manager, Multimedia Wikimedia Foundation
Fabrice Florin, 15/03/2014 00:13:
We would appreciate your advice on our upcoming research study of image load times on Media Viewer. [...] *I. Goals* The goal of this study is to determine whether or not Media Viewer is loading images fast enough for the majority of our users in most common situations.
As a typical user of the Media Viewer, I want images to load quickly, in just a few seconds, so I don't have to wait a long time to see them. [...]
Definitions: [...] *II. Questions* [...]
*III. Outcomes* [...]
I'm confused. Too many questions, too many arbitrary definitions, axioms. No falsifiability. I understand the idea of defining a minimum quality standard to respect, it might even be the only way, but it's a thicket that moreover only indirectly verifies what we're actually interested in. At its root is simple, we need to know if readers enjoy the images/media more or get annoyed and don't look at them because they're too slow. (Measuring the value they get from the media, or attach to the page in consequence hence becoming more likely to visit the project more, is less clearcut.) So maybe there is some simple check for this, if surveys don't work maybe just the total number of requests or something.
Nemo
Thanks, Nemo.
This particular study is focused on measuring the image load time, which we view as the most critical factor for providing an acceptable performance to end-users.
We will also run a separate survey to find out if end-users find this experience useful, or if they have important issues we should address. (1)
That qualitative feedback will be reviewed in conjunction with the first quantitative results, as well as discussions with our communities, in order to determine our next steps for this project.
I look forward to learning more from our users very soon!
Enjoy your weekend,
Fabrice
(1) https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards/261
On Mar 14, 2014, at 5:23 PM, Federico Leva (Nemo) nemowiki@gmail.com wrote:
I'm confused. Too many questions, too many arbitrary definitions, axioms. No falsifiability. I understand the idea of defining a minimum quality standard to respect, it might even be the only way, but it's a thicket that moreover only indirectly verifies what we're actually interested in. At its root is simple, we need to know if readers enjoy the images/media more or get annoyed and don't look at them because they're too slow. (Measuring the value they get from the media, or attach to the page in consequence hence becoming more likely to visit the project more, is less clearcut.) So maybe there is some simple check for this, if surveys don't work maybe just the total number of requests or something.
Nemo
Multimedia mailing list Multimedia@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/multimedia
_______________________________
Fabrice Florin Product Manager Wikimedia Foundation
On Fri, Mar 14, 2014 at 5:23 PM, Federico Leva (Nemo) nemowiki@gmail.comwrote:
Fabrice Florin, 15/03/2014 00:13:
We would appreciate your advice on our upcoming research study of image load times on Media Viewer. [...] *I. Goals* The goal of this study is to determine whether or not Media Viewer is loading images fast enough for the majority of our users in most common situations.
As a typical user of the Media Viewer, I want images to load quickly, in just a few seconds, so I don't have to wait a long time to see them. [...]
Definitions: [...] *II. Questions* [...]
*III. Outcomes* [...]
I'm confused. Too many questions, too many arbitrary definitions, axioms. No falsifiability. I understand the idea of defining a minimum quality standard to respect, it might even be the only way, but it's a thicket that moreover only indirectly verifies what we're actually interested in. At its root is simple, we need to know if readers enjoy the images/media more or get annoyed and don't look at them because they're too slow. (Measuring the value they get from the media, or attach to the page in consequence hence becoming more likely to visit the project more, is less clearcut.) So maybe there is some simple check for this, if surveys don't work maybe just the total number of requests or something.
Hey Federico,
I hope you'll forgive me for some shameless (but brief) thread-jacking that won't answer any of your questions but instead raise a couple of tangential points. Apologies for the derail.
I regret not being subscribed to the multimedia list (a mistake I plan to rectify immediate after sending off this e-mail) and thus not having had the chance to respond sooner. I haven't yet had a chance to review the results, but I did have a chance to closely review the instrumentation code that the Multimedia team devised for collecting these measurements, and I can tell you that it is extremely sophisticated, making use of a web performance API, the specification of which has graduated to W3C recommendation status *last month*. I don't know yet if any errors were made in the statistical sampling and aggregation of data (and I want to be careful not to suggest that there have been any), but I do want to stress that this represents a major technical achievement and that I think we will glean a lot of insight about the performance of multimedia content on Wikimedia wikis from this infrastructure for years to come. So, kudos for that!
--- Ori Livneh ori@wikimedia.org
On Fri, Mar 14, 2014 at 5:23 PM, Federico Leva (Nemo) nemowiki@gmail.comwrote:
Fabrice Florin, 15/03/2014 00:13:
We would appreciate your advice on our upcoming research study of image load times on Media Viewer. [...] *I. Goals* The goal of this study is to determine whether or not Media Viewer is loading images fast enough for the majority of our users in most common situations.
As a typical user of the Media Viewer, I want images to load quickly, in just a few seconds, so I don't have to wait a long time to see them. [...]
Definitions: [...] *II. Questions* [...]
*III. Outcomes* [...]
I'm confused. Too many questions, too many arbitrary definitions, axioms. No falsifiability. I understand the idea of defining a minimum quality standard to respect, it might even be the only way, but it's a thicket that moreover only indirectly verifies what we're actually interested in. At its root is simple, we need to know if readers enjoy the images/media more or get annoyed and don't look at them because they're too slow. (Measuring the value they get from the media, or attach to the page in consequence hence becoming more likely to visit the project more, is less clearcut.) So maybe there is some simple check for this, if surveys don't work maybe just the total number of requests or something.
Nemo
(Cross-posting to multimedia list since I think my reply there was bounced.)
Hey Federico,
I hope you'll forgive me for some shameless (but brief) thread-jacking that won't answer any of your questions but instead raise a couple of tangential points. Apologies for the derail.
I regret not being subscribed to the multimedia list (a mistake I plan to rectify immediate after sending off this e-mail) and thus not having had the chance to respond sooner. I haven't yet had a chance to review the results, but I did have a chance to closely review the instrumentation code that the Multimedia team devised for collecting these measurements, and I can tell you that it is extremely sophisticated, making use of a web performance API, the specification of which has graduated to W3C recommendation status *last month*. I don't know yet if any errors were made in the statistical sampling and aggregation of data (and I want to be careful not to suggest that there have been any), but I do want to stress that this represents a major technical achievement and that I think we will glean a lot of insight about the performance of multimedia content on Wikimedia wikis from this infrastructure for years to come. So, kudos for that!
--- Ori Livneh ori@wikimedia.org
Hi Ori,
We haven't fully analyzed the data we've already collected yet. I've done some work on that front today: https://www.mediawiki.org/wiki/Multimedia/Performance_Analysis
The graphs we have on limn at the moment are definitely very crude, we'll improve them based on the one-off study I did today.
While working on this I also found out that we needed this header: https://gerrit.wikimedia.org/r/#/c/119027/ to be able to get all the information for image loads in production. So hopefully in a few days we'll be able to extract more information in regards to file size, bandwidth experience and varnish hits vs misses.
On Sat, Mar 15, 2014 at 3:04 AM, Ori Livneh ori@wikimedia.org wrote:
On Fri, Mar 14, 2014 at 5:23 PM, Federico Leva (Nemo) nemowiki@gmail.comwrote:
Fabrice Florin, 15/03/2014 00:13:
We would appreciate your advice on our upcoming research study of image load times on Media Viewer. [...] *I. Goals* The goal of this study is to determine whether or not Media Viewer is loading images fast enough for the majority of our users in most common situations.
As a typical user of the Media Viewer, I want images to load quickly, in just a few seconds, so I don't have to wait a long time to see them. [...]
Definitions: [...] *II. Questions* [...]
*III. Outcomes* [...]
I'm confused. Too many questions, too many arbitrary definitions, axioms. No falsifiability. I understand the idea of defining a minimum quality standard to respect, it might even be the only way, but it's a thicket that moreover only indirectly verifies what we're actually interested in. At its root is simple, we need to know if readers enjoy the images/media more or get annoyed and don't look at them because they're too slow. (Measuring the value they get from the media, or attach to the page in consequence hence becoming more likely to visit the project more, is less clearcut.) So maybe there is some simple check for this, if surveys don't work maybe just the total number of requests or something.
Nemo
(Cross-posting to multimedia list since I think my reply there was bounced.)
Hey Federico,
I hope you'll forgive me for some shameless (but brief) thread-jacking that won't answer any of your questions but instead raise a couple of tangential points. Apologies for the derail.
I regret not being subscribed to the multimedia list (a mistake I plan to rectify immediate after sending off this e-mail) and thus not having had the chance to respond sooner. I haven't yet had a chance to review the results, but I did have a chance to closely review the instrumentation code that the Multimedia team devised for collecting these measurements, and I can tell you that it is extremely sophisticated, making use of a web performance API, the specification of which has graduated to W3C recommendation status *last month*. I don't know yet if any errors were made in the statistical sampling and aggregation of data (and I want to be careful not to suggest that there have been any), but I do want to stress that this represents a major technical achievement and that I think we will glean a lot of insight about the performance of multimedia content on Wikimedia wikis from this infrastructure for years to come. So, kudos for that!
Ori Livneh ori@wikimedia.org
Multimedia mailing list Multimedia@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/multimedia
Ok, thanks Ori and Gilles; I hadn't realised that the instrumentation in question had already been developed, of course it's there so it must be used. Having insight on performance not necessarily translates into having data useful for decisions, but it's nevertheless very appreciated.
Nemo
multimedia@lists.wikimedia.org