On Thu, Jan 15, 2015 at 2:30 PM, Gilles Dubuc <gilles@wikimedia.org> wrote:

Erik came up with the following suggestion on the whole thumbnails-stored-in-swift issue: would it be possible to store thumbnails in a separate Swift cluster than originals, and have the replica count setting for the thumbnail swift cluster set to 1?

The rationale being that this would solve the main issue of storage space waste without having to learn/test/deploy a new tool.

it is possible to have a separate swift cluster but I'm not sure it'd yield many benefits because e.g. each swift machine would become a "SPOF" in the sense if it is down imagescalers would have to take all hits for all thumbs stored on that machine.

To give some numbers, originals take up ~58T and thumbs ~32T while there are ~27M originals vs ~428M thumbs and from swift thumbs are requested ~6.5x more than originals.

There's more proposals here too: https://www.mediawiki.org/wiki/Talk:Requests_for_comment/Simplify_thumbnail_cache

HTH,

filippo