Hi everyone,
We would appreciate your advice on our upcoming research study of image load times on
Media Viewer.
Here are proposed goals, questions and outcomes for this study. They are presented for
discussion purposes, not as a prescriptive requirement - and will be adjusted based on
your feedback.
I. Goals
The goal of this study is to determine whether or not Media Viewer is loading images fast
enough for the majority of our users in most common situations.
As a typical user of the Media Viewer, I want images to load quickly, in just a few
seconds, so I don't have to wait a long time to see them.
Here are our recommended performance targets for image load times by connection speed, to
match user expectations on the Web:
* 1-2 seconds for a medium-size image on a fast connection
* 2-3 seconds for the same image on a medium connection
* 5-8 seconds for the same image a slow connection
If tracking connection speeds is too hard in our time-frame, we could base our performance
targets on image size instead. For example:
* 1-2 seconds for a small-size image on a medium connection
* 2-3 seconds for medium-size image on the same connection
* 5-8 seconds for large-size image on the same connection
Definitions:
* Image load time = the number of seconds from when you click on a thumbnail to when you
see the full image
* Image size: large = over 2Mb, medium = 1 to 2Mb, small = under 1Mb
* Connection speed: fast = over 256 Kbs, medium = 64 to 256 Kbs, slow = under 64 Kbs
The above numbers are for discussion purposes, and can be adjusted based on your
feedback.
II. Questions
Here are the main research questions we propose to answer about image load performance.
1. How long does it take for an image to load for the conditions below?
(image load = total time from thumbnail click to full image display)
a. by image size:
load times for large images? medium images? small images?
b. by web site:
load times for mediawiki.org? commons? enwiki? frwiki? huwiki? other sites?
c. by connection speed: (optional)
load times for fast connections? medium connections? small connections? (this may not
be feasible in our time frame)
d. by daypart: (optional)
load times for morning? afternoon? evening? night time? (to show if performance slows
during peak hours)
This question could be answered by storing the timestamp for thumbnail clicks, as well as
the timestamp for the full image display, then log the difference.
We would then prepare different bar graphs for each condition set above, with categories
on the vertical axis, and number of seconds on the horizontal axis. The graphs could be
based on data from the last 7 days.
2. How often does the image load time exceed our performance targets above?
a. by load time in a day:
number of images that load in under 1 second? in 1-2 seconds? in 2-3 seconds? … and so
on, up to 10 seconds or more
b. by load time in a week:
number of images that load in under 1 second? in 1-2 seconds? in 2-3 seconds? … and so
on, up to 10 seconds or more
This question could be answered by preparing different histograms, with number of images
on the vertical axis, and number of seconds on the horizontal axis (deciles).
III. Outcomes
To answer these questions, we plan to collect data during our upcoming pilots on different
sites in April.
Based on these pilot results, we will need to make decisions about the wider deployments
planned for May.
Here are possible outcomes from this study:
Outcome 1: Favorable - e.g.: 80% of images load quickly
Action: Go ahead with current release plan to deploy Media Viewer everywhere by default.
Scenario 2: Neutral - e.g.: 50% of images load quickly
Action: Go ahead with current release plan, but deploy Media Viewer as an opt-in feature
on wikis that don’t want it by default
Scenario 3: Unfavorable - e.g.: 20% of images load quickly
Action: Revisit release plan: consider making this opt-in everywhere — or work on faster
image load solutions.
We would be grateful for your comments on this, so we can refine our plans before we start
this study next week. Please let us know which metrics above seem most important, given
that we only have a few developer days to collect and analyze a few key metrics in coming
weeks, to determine if we are meeting our objectives. Some related links are included
below, for your convenience.
To end on a positive note, we just deployed yesterday a new version of Media Viewer that
is much faster, thanks to all the fine work from our development team. This morning, I
looked at a variety on 'non-popular' images on enwiki today, and the Media Viewer
experience was quite good overall. Most images load within the 2 second maximum which we
recommend for a ‘fast’ connection — and this was a home wifi connection. I realize this is
completely anecdotal, and not supported by hard data, so we can’t make any decisions about
it. But it makes me hopeful that we are getting close to our objectives. Even compared to
large commercial sites like Flickr, we hold up pretty well on this computer. :)
Thanks for your interest in this project.
All the best,
Fabrice
_______________________________
USEFUL LINKS
* Media Viewer Release Plan:
https://www.mediawiki.org/wiki/Multimedia/Media_Viewer/Release_Plan
* First Media Viewer Metrics:
http://multimedia-metrics.wmflabs.org/dashboards/mmv Metrics
* Media Viewer Test Page:
https://commons.wikimedia.org/wiki/Commons:Lightbox_demo
* Metrics Tasks under consideration (Mingle):
https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards?favorit…
* Next Development Cycle (Mingle):
http://ur1.ca/gtvvr
* About Media Viewer:
https://www.mediawiki.org/wiki/Multimedia/About_Media_Viewer
_______________________________
Fabrice Florin
Product Manager, Multimedia
Wikimedia Foundation
http://en.wikipedia.org/wiki/User:Fabrice_Florin_(WMF)