Hi ops,
Has the swift capacity been increased yet thanks to the new hardware? If so, could we resume the discussion of "pre"generating specific thumbnail sizes at upload time?
Media Viewer could benefit greatly from this performance-wise. As seen on this graph, the launch to all wikis affected the average considerably, since users started hitting a lot of images that didn't have Media Viewer-sized thumbnails yet: http://multimedia-metrics.wmflabs.org/dashboards/mmv#overall_network_perform...
Thumbnailing improvements are still in the works on our end, and the idea of not using swift anymore for those is definitely on the team's radar (we've started working on more modest thumbnailing improvements at this point), but if the capacity is there, we might as well improve the average image load time for our users, even if the ever-increasing swift use for thumbnail is still a problem itself.
On Mon, Apr 21, 2014 at 11:16 AM, Gilles Dubuc gilles@wikimedia.org wrote:
Hi all,
Do you guys have a rough estimate of when the new swift capacity will be installed?
Now that we're in the process of Media Viewer's launch to all users on pilot sites, the question of when we'll be able to prerender thumbnails at desired bucket sizes has come up again.
On Fri, Jul 4, 2014 at 1:03 AM, Gilles Dubuc gilles@wikimedia.org wrote:
Hi ops,
Has the swift capacity been increased yet thanks to the new hardware? If so, could we resume the discussion of "pre"generating specific thumbnail sizes at upload time?
Media Viewer could benefit greatly from this performance-wise. As seen on this graph, the launch to all wikis affected the average considerably, since users started hitting a lot of images that didn't have Media Viewer-sized thumbnails yet: http://multimedia-metrics.wmflabs.org/dashboards/mmv#overall_network_perform...
This looks pretty bad. Thanks for calling it out.
Thumbnailing improvements are still in the works on our end, and the idea
of not using swift anymore for those is definitely on the team's radar (we've started working on more modest thumbnailing improvements at this point)
May I ask: what is the range of this radar? Radar signal declines in quality as distance increases, and it is vulnerable to interference from diverse factors, like atmospheric turbulence and birds.
Nobody likes sinking time, energy and hardware on an architecture that doesn't work. Even when it's the right thing to do. I think appetite for this work might grow if some there were some definite plans afoot for moving us past it, to a setup that works.
On Sun, Jul 6, 2014 at 3:34 AM, Ori Livneh ori@wikimedia.org wrote:
On Fri, Jul 4, 2014 at 1:03 AM, Gilles Dubuc gilles@wikimedia.org wrote:
Hi ops,
Has the swift capacity been increased yet thanks to the new hardware? If so, could we resume the discussion of "pre"generating specific thumbnail sizes at upload time?
Media Viewer could benefit greatly from this performance-wise. As seen on this graph, the launch to all wikis affected the average considerably, since users started hitting a lot of images that didn't have Media Viewer-sized thumbnails yet: http://multimedia-metrics.wmflabs.org/dashboards/mmv#overall_network_perform...
This looks pretty bad. Thanks for calling it out.
I don't think it's especially bad; there is a spike after the rollout (it can be seen more clearly if you scroll down to the imagemiss stats) which lasts about five says, other than that it's just probably the effect of rolling out to new userbases which have on average much worse network conditions then the Europe/USA based ones.
If you look at wikis to which we have rolled out earlier, e.g. http://multimedia-metrics.wmflabs.org/dashboards/mmv_enwiki#overall_network_... there is no change at all.
Which is not to say the lack of pregenerated thumbnails is not a serious problem (I just don't think it got any worse recently). Comparing the global imagehit and imagemiss stats, the lack of pregeneration affects about 20% of the requests, and costs about 730ms (an extra 85% loading time) for the median user.
Hi Gilles,
On 04 Jul 2014, at 10:03, Gilles Dubuc gilles@wikimedia.org wrote:
Hi ops,
Has the swift capacity been increased yet thanks to the new hardware? If so, could we resume the discussion of "pre"generating specific thumbnail sizes at upload time?
Media Viewer could benefit greatly from this performance-wise. As seen on this graph, the launch to all wikis affected the average considerably, since users started hitting a lot of images that didn't have Media Viewer-sized thumbnails yet: http://multimedia-metrics.wmflabs.org/dashboards/mmv#overall_network_perform...
Yes, our Swift capacity expansion has completed. Although the capacity expansion had been planned with reduced storage for thumbs in mind for the future, work that hasn’t started yet, we should have space for this available right now and it shouldn’t be problematic provided that we monitor it well. So please go ahead with this for newly uploaded images - we’ll see how it goes.
Filippo will look into whether Swift’s TTL feature is usable for us these days; hopefully we can use it to reduce storage of unused thumbs / thumb sizes until we move them out of Swift completely.
Thumbnailing improvements are still in the works on our end, and the idea of not using swift anymore for those is definitely on the team's radar (we've started working on more modest thumbnailing improvements at this point), but if the capacity is there, we might as well improve the average image load time for our users, even if the ever-increasing swift use for thumbnail is still a problem itself.
Yeah. As you know there’s a fair amount of desire to change the way thumb handling & storage works, support more variants (sizes, quality), etc by various teams. I’m starting to think it would be good for us to organise a sprint on this very topic, get the multimedia team, relevant Ops people and some other interested developers from other teams and really dive into these problems.
— Mark Bergsma mark@wikimedia.org Lead Operations Architect Director of Technical Operations Wikimedia Foundation
multimedia@lists.wikimedia.org