_don't consider the upload complete_ until those are done! a web uploader or API-using bot should probably wait until it's done before uploading the next file, for instance...

On 04/30/2014 12:51 PM, Brion Vibber wrote: Would it not suffice to just produce *one* scaled down version (ie. 2048px) which the real-time scaler can use to produce the thumbs? Regards, -- Erwin Dokter _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Another point about picking the "one true bucket list": currently Media Viewer's buckets have been picked based on the most common screen resolutions, because Media Viewer tries to always use the entire width of the screen to display the image, so trying to achieve a 1-to-1 pixel correspondence makes sense, because it should give the sharpest result possible to the average user. However, sticking to that approach will likely introduce a cost. As I've just mentioned, we will probably need to generate more than one of the high buckets based on the original, in order to avoid resizing artifacts. On the other hand, we could decide that the unified bucket list shouldn't be based on screen resolutions (after all the full width display scenario experienced in Media Viewer might be the exception, and the buckets will be for everything mediawiki) and instead would progress by powers of 2. Then creating a given bucket could always be done without resizing artifacts, based on the bucket above the current one. This should provide the biggest savings possible in image scaling time to generate thumbnail buckets. To illustrate with an example, the bucket list could be: 256, 512, 1024, 2048, 4096. The 4096 bucket would be generated first, based on the original, then 2048 would be generated based on 4096, then 1024 based on 2048, etc. The big downside is that there's less progression in the 1000-3000 range (4 buckets in the Media Viewer resolution-oriented strategy, 2 buckets here) where the majority of devices currently are. If I take a test image as an example ( https://commons.wikimedia.org/wiki/File:Swallow_flying_drinking.jpg), the file size progression is quite different between the screen resolution buckets and the geometric (powers of 2) buckets: - screen resolution buckets 320 11.7kb 640 17kb 800 37.9kb 1024 58kb 1280 89.5kb 1920 218.9kb 2560 324.6kb 2880 421.5kb - geometric buckets 256 9.4kb 512 20kb 1024 58kb 2048 253.1kb 4096 test image is smaller than 4096 It seems like it's not ideal that a screen resolution slightly above 1024 would suddenly need to download an image 5 times as heavy, for not that many extra pixels on the actual screen. A similar thing can be said for the screen resolution progression, where the file size more than doubles between 1280 and 1920. We could probably use at least an extra step between those two if we use screen resolution buckets, like 1366 and/or 1440. I think that the issue of buckets between 1000 and 3000 is tricky, it's going to be difficult to avoid generating them based on the original while not getting visual artifacts. Maybe we can get away with generating 1280 (and possibly 1366, 1440) based on 2048, the distance between the two guaranteeing that the quality issues will be negligible. We definitely can't generate a 1920 based on a 2048 thumbnail, though, otherwise artifacts on thin lines will look awful. A mixed progression like this might be the best of both worlds, if we confirm that between 1024 and 2048 the resizing is artifact-free enough: 256, 512, 1024, 1280, 1366, 1440, 2048, 4096 where 2048 is generated based on 4096, 1024, 1280, 1366, 1440 are generated based on 2048, 512 based on 1024, 256 based on 512. If for example the image width is between 1440 and 2048, then 1024, 1280, 1366, 1440 would be generated based on the original. That's fine performance-wise, since the original is small. Something that might also be useful to generate is a thumbnail of the same size as the original if original < 4096 (or whatever the highest bucket is). Currently we seem to block generating such a thumbnail, but the difference in file size is huge. For the test image mentioned above, which is 3002 pixels wide, the original is 3.69MB, while a thumbnail of the same size would be 465kb. For the benefit of retina displays that are 2560/2880, displaying a thumbnail of the same size as a 3002 original would definitely be better than the highest available bucket (2048). All of this is benchmark-worthy anyway, I might be splitting hair looking for powers of two if rendering a bucket chain (each bucket generated based on the next one) isn't that much faster than generating all buckets based on the biggest bucket. On Thu, May 1, 2014 at 12:54 PM, Gilles Dubuc &lt;gilles(a)wikimedia.org&gt;wrote;wrote:

An extremely crude benchmark on our multimedia labs instance, still using the same test image: original -> 3002 (original size) 0m0.268s original -> 2048 0m1.344s original -> 1024 0m0.856s original -> 512 0m0.740s original -> 256 0m0.660s 2048 -> 1024 0m0.444s 2048 -> 512 0m0.332s 2048 -> 256 0m0.284s 1024 -> 512 0m0.112s 512 -> 256 0m0.040s Which confirms that chaining instead of generating all thumbnails based on the biggest bucket saves a significant amount of processing time. It's definitely in the same order or magnitude as the savings achieved by going from original as the source to the biggest bucket as the source. It's also worth noting that generating the thumbnail of the same size as the original is relatively cheap. Using it as the source for the 2048 image doesn't save that much time, though: 0m1.252s (for 3002 -> 2048). And here's a side-by-side comparison of these images generated with chaining and images that come from our regular image scalers: https://dl.dropboxusercontent.com/u/109867/imagickchaining/index.html Try to guess which is which before inspecting the page for the answer :) On Thu, May 1, 2014 at 4:02 PM, Gilles Dubuc &lt;gilles(a)wikimedia.org&gt; wrote: high be biggest the the between while based issues based same which same 2560/2880, definitely based uploader the visual the It could want compared test we out a biggest https://www.mediawiki.org/wiki/Requests_for_comment/Standardized_thumbnails… propose a single bucket list option instead of multiple use be they available the thumbnail current sizes wrote: _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Buttons is French: Suiv. -> Make it English

I wonder if the mip-mapping approach could somehow be combined with tiles? If we want proper zooming for large images, we will have to split them up into tiles of various sizes, and serve only the tiles for the visible portion when the user zooms on a small section of the image. Splitting up an image is a fast operation, so maybe it could be done on the fly (with caching for a small subset based on traffic), in which case having a chain of scaled versions of the image would take care of the zooming use case as well.

I wonder if the mip-mapping approach could somehow be combined with tiles? If we want proper zooming for large images, we will have to split them up into tiles of various sizes, and serve only the tiles for the visible portion when the user zooms on a small section of the image. Splitting up an image is a fast operation, so maybe it could be done on the fly (with caching for a small subset based on traffic), in which case having a chain of scaled versions of the image would take care of the zooming use case as well.

After taking a closer look at what commands run exactly in production, it turns out that I probably applied the IM parameters in the wrong order when I put together the survey (order matters, particularly for sharpening). I'll regenerate the images and make another (hopefully better) survey that will compare the status quo, chained thumbnails and single thumbnail reference. On Mon, May 5, 2014 at 11:04 AM, Gilles Dubuc &lt;gilles(a)wikimedia.org&gt;wrote;wrote: