Hi everyone,
Bryan Tong Minh has been doing some great work on VipsScaler extension, but there hasn't been a lot of recent chatter on the mailing list about it. Since we're somewhat close to deploying something on the cluster, it probably warrants at least an email.
First, some background. Currently, we use ImageMagick for scaling images on Wikimedia websites. VIPS (http://www.vips.ecs.soton.ac.uk/ ) is an alternative image processing system that has a couple advantages over ImageMagick for our purposes: 1) it's faster, and 2) it's way more efficient with memory. The latter means that we can allow for much larger images with VIPS than we can currently allow using ImageMagick. For example, we currently disallow PNG images greater than 12.5 MP.
Changing image scalers may introduce bugs, so we're proceeding cautiously. We have a three step plan: Step 1: Deploy VipsScaler extension, but only use it for PNGs over 12.5MP and/or TIFFs (which currently generate errors). Let this sit in production a while, fixing any bugs we find with this configuration. Step 2: Deploy a comparison tool (to be written) that allows anyone to rescale an image with VIPS, and compare it to the ImageMagick version. Allow more time for testing and fixing. Step 3: Switch to VIPS for everything
If you're interested in following progress, there's a couple places you can look: * VipsScaler tracking bug: https://bugzilla.wikimedia.org/28135 * Follow changes in SVN: https://www.mediawiki.org/wiki/Special:Code/MediaWiki?path=%2Ftrunk%2Fextens...
Step 1 could go out as early as next week, pending final review of the extension. If anyone is interested in helping accelerate this, the comparison tool in step 2 is only just started (it's the SpecialVipsTest special page code), and Bryan is only working on it when he has time (which isn't right now).
When we're ready to schedule this, we'll update the calendar here: https://wikitech.wikimedia.org/view/Software_deployments
Rob
On Wed, Nov 2, 2011 at 10:08 PM, Rob Lanphier robla@wikimedia.org wrote:
Step 1: Deploy VipsScaler extension, but only use it for PNGs over 12.5MP and/or TIFFs (which currently generate errors). Let this sit in production a while, fixing any bugs we find with this configuration.
I should note that PagedTiffHandler currently does support vips, but only unconditionally; we can't choose to only scale images with vips that are currently too large. If tiff support is to be enabled in the first step, then PagedTiffHandler should somehow be interfacing with VipsHandler, which does allow conditional scaling.
Bryan
On 03/11/11 08:36, Bryan Tong Minh wrote:
On Wed, Nov 2, 2011 at 10:08 PM, Rob Lanphier robla@wikimedia.org wrote:
Step 1: Deploy VipsScaler extension, but only use it for PNGs over 12.5MP and/or TIFFs (which currently generate errors). Let this sit in production a while, fixing any bugs we find with this configuration.
I should note that PagedTiffHandler currently does support vips, but only unconditionally; we can't choose to only scale images with vips that are currently too large. If tiff support is to be enabled in the first step, then PagedTiffHandler should somehow be interfacing with VipsHandler, which does allow conditional scaling.
I think it's reasonable to include all of the TIFFs in the initial deployment, as a beta test. TIFFs are rarely used, and the people who upload them are relatively technically capable, so the impact of any issues would be limited.
-- Tim Starling
On Wed, Nov 2, 2011 at 10:08 PM, Rob Lanphier robla@wikimedia.org wrote:
Step 1: Deploy VipsScaler extension, but only use it for PNGs over 12.5MP and/or TIFFs (which currently generate errors). Let this sit in production a while, fixing any bugs we find with this configuration.
I should note that PagedTiffHandler currently does support vips, but only unconditionally; we can't choose to only scale images with vips that are currently too large. If tiff support is to be enabled in the first step, then PagedTiffHandler should somehow be interfacing with VipsHandler, which does allow conditional scaling.
It should not be that much of a problem to add conditional use of vips to PagedTiffHandler. But I need to understand this a bit better :) So images that are bigger than a given threshold size (say, $wgPagedTiffHandlerUseVipsMinSize) should use vips as a scaler, below that threshold imagemagick should be used? Or can this value be determined dynamically from the vips handler?
Markus (mglaser)
_______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Rob Lanphier wrote:
Bryan Tong Minh has been doing some great work on VipsScaler extension, but there hasn't been a lot of recent chatter on the mailing list about it. Since we're somewhat close to deploying something on the cluster, it probably warrants at least an email.
The name kinda sucks. Does VIPS stand for something? I only mention this as it's possible more people would be talking about it if they understood what it was. A name like ImageScaler or something is much clearer.
In any case, it sounds very neat. :-) And the VIPS dev team is using MediaWiki, which is cool.
For example, we currently disallow PNG images greater than 12.5 MP.
Are you sure about this? I recall a configurable 12.5 MP limit on rendering of JPGs, but does that also apply to PNGs? And "disallow" is a bit ambiguous here. They just won't render / have thumbnails created, they aren't blocked from being uploaded, no?
Step 1: Deploy VipsScaler extension, but only use it for PNGs over 12.5MP and/or TIFFs (which currently generate errors). Let this sit in production a while, fixing any bugs we find with this configuration.
No deployment to a labs or test site first?
MZMcBride
On Wed, Nov 2, 2011 at 5:00 PM, MZMcBride z@mzmcbride.com wrote:
Rob Lanphier wrote:
For example, we currently disallow PNG images greater than 12.5 MP.
Are you sure about this? I recall a configurable 12.5 MP limit on rendering of JPGs, but does that also apply to PNGs?
Actually it's JPEGs that we *exclude from* the megapixel size limit for thumbnailing, because ImageMagick can scale the JPEGs more efficiently by decoding them to a lower pre-decimated resolution.
-- brion
MZMcBride wrote:
For example, we currently disallow PNG images greater than 12.5 MP.
Are you sure about this? I recall a configurable 12.5 MP limit on rendering of JPGs, but does that also apply to PNGs? And "disallow" is a bit ambiguous here. They just won't render / have thumbnails created, they aren't blocked from being uploaded, no?
It's the other way. JPEGs are scaled, because they are decompressed "in a pipe" bug pngs are not. We have pngds program to deal with that, but it was never deployed.
On 04/11/11 09:56, Platonides wrote:
MZMcBride wrote:
For example, we currently disallow PNG images greater than 12.5 MP.
Are you sure about this? I recall a configurable 12.5 MP limit on rendering of JPGs, but does that also apply to PNGs? And "disallow" is a bit ambiguous here. They just won't render / have thumbnails created, they aren't blocked from being uploaded, no?
It's the other way. JPEGs are scaled, because they are decompressed "in a pipe" bug pngs are not. We have pngds program to deal with that, but it was never deployed.
VIPS is a more complete solution to the problem than pngds. Note that VipsScaler and pngds were written by the same person.
-- Tim Starling
On Sat, Nov 5, 2011 at 2:59 AM, Tim Starling tstarling@wikimedia.org wrote:
VIPS is a more complete solution to the problem than pngds. Note that VipsScaler and pngds were written by the same person.
-- Tim Starling
Last time I checked pngds on some very large PNGs its results were of noticeably lower quality and it was generally slower, though surely less memory-hungry.
--vvv
wikitech-l@lists.wikimedia.org