Hi everyone,
We would appreciate your advice on our upcoming research study of image load times on Media Viewer.
Here are proposed goals, questions and outcomes for this study. They are presented for discussion purposes, not as a prescriptive requirement - and will be adjusted based on your feedback.
I. Goals The goal of this study is to determine whether or not Media Viewer is loading images fast enough for the majority of our users in most common situations.
As a typical user of the Media Viewer, I want images to load quickly, in just a few seconds, so I don't have to wait a long time to see them.
Here are our recommended performance targets for image load times by connection speed, to match user expectations on the Web: * 1-2 seconds for a medium-size image on a fast connection * 2-3 seconds for the same image on a medium connection * 5-8 seconds for the same image a slow connection
If tracking connection speeds is too hard in our time-frame, we could base our performance targets on image size instead. For example: * 1-2 seconds for a small-size image on a medium connection * 2-3 seconds for medium-size image on the same connection * 5-8 seconds for large-size image on the same connection
Definitions: * Image load time = the number of seconds from when you click on a thumbnail to when you see the full image * Image size: large = over 2Mb, medium = 1 to 2Mb, small = under 1Mb * Connection speed: fast = over 256 Kbs, medium = 64 to 256 Kbs, slow = under 64 Kbs
The above numbers are for discussion purposes, and can be adjusted based on your feedback.
II. Questions Here are the main research questions we propose to answer about image load performance.
1. How long does it take for an image to load for the conditions below? (image load = total time from thumbnail click to full image display)
a. by image size: load times for large images? medium images? small images?
b. by web site: load times for mediawiki.org? commons? enwiki? frwiki? huwiki? other sites?
c. by connection speed: (optional) load times for fast connections? medium connections? small connections? (this may not be feasible in our time frame)
d. by daypart: (optional) load times for morning? afternoon? evening? night time? (to show if performance slows during peak hours)
This question could be answered by storing the timestamp for thumbnail clicks, as well as the timestamp for the full image display, then log the difference.
We would then prepare different bar graphs for each condition set above, with categories on the vertical axis, and number of seconds on the horizontal axis. The graphs could be based on data from the last 7 days.
2. How often does the image load time exceed our performance targets above?
a. by load time in a day: number of images that load in under 1 second? in 1-2 seconds? in 2-3 seconds? … and so on, up to 10 seconds or more
b. by load time in a week: number of images that load in under 1 second? in 1-2 seconds? in 2-3 seconds? … and so on, up to 10 seconds or more
This question could be answered by preparing different histograms, with number of images on the vertical axis, and number of seconds on the horizontal axis (deciles).
III. Outcomes To answer these questions, we plan to collect data during our upcoming pilots on different sites in April.
Based on these pilot results, we will need to make decisions about the wider deployments planned for May.
Here are possible outcomes from this study:
Outcome 1: Favorable - e.g.: 80% of images load quickly Action: Go ahead with current release plan to deploy Media Viewer everywhere by default.
Scenario 2: Neutral - e.g.: 50% of images load quickly Action: Go ahead with current release plan, but deploy Media Viewer as an opt-in feature on wikis that don’t want it by default
Scenario 3: Unfavorable - e.g.: 20% of images load quickly Action: Revisit release plan: consider making this opt-in everywhere — or work on faster image load solutions.
We would be grateful for your comments on this, so we can refine our plans before we start this study next week. Please let us know which metrics above seem most important, given that we only have a few developer days to collect and analyze a few key metrics in coming weeks, to determine if we are meeting our objectives. Some related links are included below, for your convenience.
To end on a positive note, we just deployed yesterday a new version of Media Viewer that is much faster, thanks to all the fine work from our development team. This morning, I looked at a variety on 'non-popular' images on enwiki today, and the Media Viewer experience was quite good overall. Most images load within the 2 second maximum which we recommend for a ‘fast’ connection — and this was a home wifi connection. I realize this is completely anecdotal, and not supported by hard data, so we can’t make any decisions about it. But it makes me hopeful that we are getting close to our objectives. Even compared to large commercial sites like Flickr, we hold up pretty well on this computer. :)
Thanks for your interest in this project.
All the best,
Fabrice
_______________________________
USEFUL LINKS
* Media Viewer Release Plan: https://www.mediawiki.org/wiki/Multimedia/Media_Viewer/Release_Plan
* First Media Viewer Metrics: http://multimedia-metrics.wmflabs.org/dashboards/mmv Metrics
* Media Viewer Test Page: https://commons.wikimedia.org/wiki/Commons:Lightbox_demo
* Metrics Tasks under consideration (Mingle): https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards?favorite...
* Next Development Cycle (Mingle): http://ur1.ca/gtvvr
* About Media Viewer: https://www.mediawiki.org/wiki/Multimedia/About_Media_Viewer
_______________________________
Fabrice Florin Product Manager, Multimedia Wikimedia Foundation