On Fri, Jan 9, 2015 at 9:46 AM, Gilles Dubuc <gilles@wikimedia.org> wrote:
The bottom line is that the thumbnail prerendering provided insignificant performance gains for this set of sizes. Infrequently requested thumbnails is the main problem, not the fact that they are rendered on the fly the first time they are requested.

It seems like the only way to increase image serving performance in our current setup is to increase the expiry value in Varnish and/or increase Varnish capacity. Right now 17% of image requests in Media Viewer are Varnish misses, and 99.5% of those are pulling an existing thumbnail from Swift. Varnish misses are twice as slow as hits on average.

FWIW here's what swift sees for time to first byte on HTTP GET in which it returned 200 (99% percentile and average)

https://graphite.wikimedia.org/render/?width=672&height=381&_salt=1421056180.408&from=-24hours&target=averageSeries(swift.eqiad-prod.ms-fe*.proxy-server.object.GET.200.first-byte.timing.99percentile)&target=averageSeries(swift.eqiad-prod.ms-fe*.proxy-server.object.GET.200.first-byte.timing.mean)