The Mediawiki 1.18 image rotation bug on Commons and on all Wikimedia projects
1 - Bug or feature ? It is a bug. 2 - The human bug 3 - The technical bug 4 - Unexhaustive list of related talks
1 - Bug or feature ? It is a bug.
Look at http://en.wikipedia.org/w/index.php?title=Arameans&oldid=463282677 : both pictures File:Aramean funeral stele Louvre AO3026.jpg and File:Si Gabbor funeral stele Louvre AO3027.jpg are tilted.
It is somehow intentional, because it seems that the devs have suddenly decided that the exif orientation tag should be taken into account, while in the past users used had to use other ways to define image orientation.
But even if it is intentional, we should call it a bug, because it is annoying to a lot of readers and uploaders whose pictures have been OK sometimes for years, and without warning they must suddenly change the orientation of their uploaded pictures. What about the pictures whose uploaders are no longer active ?
So I hope everybody agrees that it is a bug.
2 - The human bug
I think the Wikimedia Foundation should present officially its excuses to the readers and active users annoyed by the bug. The excuses could be linked from the rotatebot template, so that the concerned users could read them.
The devs should find out what went wrong in the decision process to implement the 1.18 version, and try to find preventive measures so that big problems of this size do not occur again when a version upgrade is done. Is it really OK not to consult the Commons community before changing a picture-related feature ?
3 - The technical bug : deadline
A lot of people should be thanked for having spared no energy to find the first steps toward solutions to the bug. A lot has been done. In particular a lot has been done to provide users easy access to a bot, called "rotatebot" which rotates pictures when needed. A lot of users have spent time tagging pictures with a "rotate" template, which calls the bot for help. Really a lot of people. The bot is busy, and the bot should be thanked, if it had brains to understand what "thank you" means.
Despite all of that, despite the fact that the bot's speed was lately increased, we are still lacking a systematic solution which would correct all wrongly rotated pictures and a deadline.
Let us stop asking users to individually tag every wrong picture! Let us have some developers create a tool to find wrong pictures and rotate them back to their original orientation!
We need a deadline. We need to be able to say, In X month's time, all pictures will be back to normal.
4 - Unexhaustive list of related talks
http://commons.wikimedia.org/wiki/Commons:Village_pump/Archive/2011/10#Autor... http://commons.wikimedia.org/wiki/Commons:Village_pump/Archive/2011/10#Autor... http://commons.wikimedia.org/wiki/Commons:Village_pump/Archive/2011/10#Rotat... http://commons.wikimedia.org/wiki/Commons:Village_pump/Archive/2011/10#probl... http://commons.wikimedia.org/wiki/Commons:Village_pump/Archive/2011/10#Rotat... http://commons.wikimedia.org/wiki/Commons:Village_pump/Archive/2011/11#New_a... http://commons.wikimedia.org/wiki/Commons:Village_pump/Archive/2011/11#Wrong... http://commons.wikimedia.org/wiki/Commons:Village_pump/Archive/2011/11#.22Re... http://commons.wikimedia.org/wiki/Commons:Village_pump/Archive/2011/12#Direc... http://commons.wikimedia.org/wiki/Commons:Bistro#Monast.C3.A8re_Andronikov_:... http://commons.wikimedia.org/wiki/Commons:Bistro#Monde_.C3.A0_l.27envers http://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Le_Bistro/5_d%C3%A9cembre_2011#P... http://commons.wikimedia.org/wiki/Commons:Village_pump#Image_rotation_-_I_am... http://commons.wikimedia.org/wiki/Commons:Bots/Work_requests#Maintenance_cat... http://commons.wikimedia.org/wiki/Commons_talk:Rotation http://commons.wikimedia.org/wiki/User_talk:Rotatebot#Rotation_on_Wikipedia
It is unexhaustive because I did not check Commons' help desk, nor every Wikipedia language version.
On Tue, Dec 13, 2011 at 12:36 AM, Teofilo teofilowiki@gmail.com wrote:
1 - Bug or feature ? It is a bug. ... snip ... It is somehow intentional, because it seems that the devs have suddenly decided that the exif orientation tag should be taken into account, while in the past users used had to use other ways to define image orientation.
It's been a requested feature for a while, Someone finally got around to writing it (I believe it needed the Improved metadata handling backend first) and implementing it, It wasn't a sudden "oh lets write this and enable it in one day thing", a lot of work went into it and subsequent testing.
But even if it is intentional, we should call it a bug, because it is annoying to a lot of readers and uploaders whose pictures have been OK sometimes for years, and without warning they must suddenly change the orientation of their uploaded pictures. What about the pictures whose uploaders are no longer active ?
So I hope everybody agrees that it is a bug.
The bug I see is software people used to edit these images didn't fix the files metadata itself, thus in the end creating this situation
2 - The human bug
I think the Wikimedia Foundation should present officially its excuses to the readers and active users annoyed by the bug. The excuses could be linked from the rotatebot template, so that the concerned users could read them.
Excuses? The reasons why it's "broken" have been posted in many places, Last I checked the said template wasn't protected so anyone could and pointers to about why its happening.
The devs should find out what went wrong in the decision process to implement the 1.18 version, and try to find preventive measures so that big problems of this size do not occur again when a version upgrade is done. Is it really OK not to consult the Commons community before changing a picture-related feature ?
Nothing much went wrong in the planning of this feature, The metadata backend was improved, the rotation feature was written, the feature was tested (and i'm aware of this because I did test it) and the feature did work as intended.
And why should commons be notified when a MediaWiki core feature is written, why not ja.wikipedia or en.wikinews? just because commons is a end user of the software doesn't make it all that special, While yes the choice to deploy it to the cluster could have been handled differently it worked from all the testing that was performed (and the issues that were found from the testing were fixed before it was pushed out).
Had more end users actually bothered to test the pre release(s) when they were staged on test. and test2.wikipedia, "issues" like this might had stood out more prominently so that its feature could have been considered after being tested on a wider scale.
3 - The technical bug : deadline ...snip... Let us stop asking users to individually tag every wrong picture! Let us have some developers create a tool to find wrong pictures and rotate them back to their original orientation!
I believe that can be done quiet easily with a DB query, Then it's just a matter of fixing the metadata attached in the file compared to actually re-rotating them again.
We need a deadline. We need to be able to say, In X month's time, all pictures will be back to normal.
A time line like that can't be given since there aren't plans to turn the feature off from my understanding, So this will conciebly be fixed when RotateBot fixes up the meta data on the files, Someone else does it, or a extension/feature is written so humans have a interface on-wiki to manually rotate the files to how they should be.
-Peachey, Signing off on what is now a new day.
The unrepentant attitude expressed above by K. Peachey increases the need for clear excuses from the Wikimedia Foundation, expressing clearly that something has gone wrong in the decision process, and that the people who think the relationship between users-community and developers the way K. Peachey is thinking, are mistaken. I don't want to address every single untruth included in K. Peachey's message. Let's say that when pictures are concerned, the input of the Commons community is useful, as is useful the input of the Georgian wikipedia when a Georgian-language-related feature is concerned. Let's say again that when users have been allowed for years - FOR YEARS - to upload pictures without concern for the exif orientation tag, revoking this allowance without prior warning is a breach of trust. And anyway, this is no reason to suddenly annoy readers, who are third parties in this developer-uploader misunderstanding and absence of dialogue. A Deadline is possible of course. All it needs is the political will from the Wikimedia Foundation management to impose a deadline to the devs.
On 12 December 2011 15:26, K. Peachey p858snake@gmail.com wrote:
It's been a requested feature for a while, Someone finally got around to writing it (I believe it needed the Improved metadata handling backend first) and implementing it, It wasn't a sudden "oh lets write this and enable it in one day thing", a lot of work went into it and subsequent testing.
* How many existing uploads, used on the wikis, were previously wrongly rotated and were fixed by the feature? * How many existing uploads, used on the wikis, were previously correctly rotated and were messed up by the feature?
i.e., was there strong reason to apply it to past images, not just new ones?
- d.
I am unable to find precise answers to your questions. But the scope of the phenomenon can be somehow understood with the following data which hint that today, the demand for rotation service has increased about 56-fold compared to June 2011. But I am unable to say how long the present high demand will last. And we must think about the unused pictures or pictures used on small projects which may require rotation but which people may be not be going to find so soon. Let alone the cases when readers find that something is wrong but are too shy to say it.
As of 24-30 June (7 days) Rotatebot was requested to rotate about 250 files in 7 days (1)
As of now, Rotatebot is handling about 250 files in 3 hours (2) (which means (24/3)*7*250 = 56*250 in 7 days)
(1) http://commons.wikimedia.org/w/index.php?title=Special:Contributions&off... (2) http://commons.wikimedia.org/w/index.php?title=Special:Contributions&lim...
Le 12 décembre 2011 16:55, David Gerard dgerard@gmail.com a écrit :
- How many existing uploads, used on the wikis, were previously
wrongly rotated and were fixed by the feature?
- How many existing uploads, used on the wikis, were previously
correctly rotated and were messed up by the feature?
i.e., was there strong reason to apply it to past images, not just new ones?
On Mon, Dec 12, 2011 at 7:55 AM, David Gerard dgerard@gmail.com wrote:
- How many existing uploads, used on the wikis, were previously
wrongly rotated and were fixed by the feature?
- How many existing uploads, used on the wikis, were previously
correctly rotated and were messed up by the feature?
As far as I understand the issue, and others can jump and correct me if I'm getting it wrong:
Technically, nothing was "messed up" by the feature. Rather, the software previously did not take EXIF rotation into account, and some images had incorrect EXIF rotation information to begin with. Those images are now shown in an incorrect rotation to the user, because the incorrect EXIF rotation info is being evaluated.
It's important to understand this, because it means that those images have been causing problems for re-users all along. If you open those images with modern image editing/viewing software, they will either be automatically rotated, or you'll be prompted by the software whether to apply the rotation noted in the EXIF tag.
The situation has been significantly exacerbated by a recent need to purge old thumbnails to free up diskspace.
So, while the cleanup that's happening now is very frustrating (and I definitely agree we could have anticipated and communicated this better), it's a cleanup that's long overdue. (Either by stripping EXIF info from files altogether, or by ensuring that the rotation of the image matches the one in the metadata.)
Is there more that we can do at the present time to help?
Thanks, Erik
On 12 December 2011 18:18, Erik Moeller erik@wikimedia.org wrote:
On Mon, Dec 12, 2011 at 7:55 AM, David Gerard dgerard@gmail.com wrote:
- How many existing uploads, used on the wikis, were previously
wrongly rotated and were fixed by the feature?
- How many existing uploads, used on the wikis, were previously
correctly rotated and were messed up by the feature?
As far as I understand the issue, and others can jump and correct me if I'm getting it wrong:
Technically, nothing was "messed up" by the feature. Rather, the software previously did not take EXIF rotation into account, and some images had incorrect EXIF rotation information to begin with. Those images are now shown in an incorrect rotation to the user, because the incorrect EXIF rotation info is being evaluated.
That's a big technicality. Surely the most important thing is how the images display to users? There were right before and now they aren't. That may not be technically messed up, but it is messed up in reality.
It's important to understand this, because it means that those images have been causing problems for re-users all along. If you open those images with modern image editing/viewing software, they will either be automatically rotated, or you'll be prompted by the software whether to apply the rotation noted in the EXIF tag.
Indeed, it's good to get these images fixed, but surely it would have been better to fix them rather than just break the workaround that was stopping people noticing they were broken?
The situation has been significantly exacerbated by a recent need to purge old thumbnails to free up diskspace.
How big a contributing factor has that been? As I understand it, only thumbnails of unused images were purged. People (including me) have been stumbling over incorrect images in articles - have they just been unlucky and the thumbnail happened to expire at the wrong time?
So, while the cleanup that's happening now is very frustrating (and I definitely agree we could have anticipated and communicated this better), it's a cleanup that's long overdue. (Either by stripping EXIF info from files altogether, or by ensuring that the rotation of the image matches the one in the metadata.)
Is there more that we can do at the present time to help?
I think, at the moment, the most useful thing would be to automate finding the broken images (basically, it's all images uploaded before the feature was introduced that have a non-zero EXIF rotation).
On 12 December 2011 18:18, Erik Moeller erik@wikimedia.org wrote:
Technically, nothing was "messed up" by the feature. Rather, the software previously did not take EXIF rotation into account, and some images had incorrect EXIF rotation information to begin with. Those images are now shown in an incorrect rotation to the user, because the incorrect EXIF rotation info is being evaluated.
That's ridiculous misuse of words. What was messed up was the presentation of images that were already displayed correctly.
It is entirely unclear to me why you appear to be evading rather than answering a fairly simple and straightforward question:
How many images used in the wikis had the pages they were on messed up by this?
- d.
On Mon, Dec 12, 2011 at 12:12 PM, David Gerard dgerard@gmail.com wrote:
What was messed up was the presentation of images that were already displayed correctly.
Well, technically, they were displayed incorrectly. ;-) The image told the software "Please rotate me", and the software didn't. But the image would tell any other software the same thing, causing pain for re-users. So it was definitely an issue that needed to get resolved, one way or another. I don't know off-hand how many images are affected (the estimate on Commons is about 50,000, but I don't know what that's based on).
The thing is, we've always gotten "drive-by" uploads by users who didn't bother to fix any rotation issues with their images after upload, and so we can't just go back and strip EXIF info from all old files, because some old files were fixed by the change. It looks to me like the only sensible response is human review followed by rotation of images that need to be fixed -- which is precisely what's happening, with a bot performing rotations as needed.
I've asked Rob Lanphier to look at this as well and determine if an additional response is needed; if you think there's more we can/should do to help, please let him know.
The best place for further discussion of this issue is: http://commons.wikimedia.org/wiki/Commons_talk:Rotation
On Mon, Dec 12, 2011 at 7:48 PM, Erik Moeller erik@wikimedia.org wrote:
The best place for further discussion of this issue is: http://commons.wikimedia.org/wiki/Commons_talk:Rotation
And, lots more discussions here as well: http://commons.wikimedia.org/wiki/Commons:Bots/Work_requests#Maintenance_cat...
If I interpret that discussion correctly, the number of globally used files that were affected is estimated to be about 20,000, with an additional 35,000 files that weren't globally used, based on analysis of the image metadata dumps.
A thought to those posting in this thread (especially some of the earlier posts):
What effect would a less aggressive tone have had? Would you have been more likely to convince your audience? less likely to alienate people?
This list often has too high a heat:light ratio. You can help fix this.
On Mon, Dec 12, 2011 at 9:52 PM, David Richfield davidrichfield@gmail.com wrote:
What effect would a less aggressive tone have had? Would you have been more likely to convince your audience? less likely to alienate people?
It's a fair point. I think part of the problem is that people are feeling that reasonable, calm, friendly inquiries are likely to be ignored and "making noise" is necessary to get attention. I want to make sure we do our best to respond to reasonable inquiries in a timely manner, and would ask all WMF staff and contractors to help me in that regard.
In general, if you feel that an engineering issue merits escalation, never hesitate to email me directly and, unless I'm totally swamped, I'll try to help. There are other folks whose job it is to help with triaging, like Mark Hershberger (mah at wikimedia dot org) and Sumana Harihareswara (sumanah at wikimedia dot org, especially for things like patch review), and of course you can also contact any of the engineering directors for tech issues, raise them on IRC, on Bugzilla, etc.
It's true that sometimes people complaining loudly helps us to take an issue more seriously, but ideally that shouldn't be necessary and our processes should work to understand what's causing pain and what isn't.
On 13/12/11 02:55, David Gerard wrote:
On 12 December 2011 15:26, K. Peachey p858snake@gmail.com wrote:
It's been a requested feature for a while, Someone finally got around to writing it (I believe it needed the Improved metadata handling backend first) and implementing it, It wasn't a sudden "oh lets write this and enable it in one day thing", a lot of work went into it and subsequent testing.
- How many existing uploads, used on the wikis, were previously
wrongly rotated and were fixed by the feature?
- How many existing uploads, used on the wikis, were previously
correctly rotated and were messed up by the feature?
i.e., was there strong reason to apply it to past images, not just new ones?
Such statistics were never gathered. I was told by the developers involved that existing images with EXIF rotation would be very rare and that most of them would be fixed by this feature, and I didn't challenge that.
I think it's too early to focus on recriminations, we risk distracting people from actually fixing the issue.
-- Tim Starling
On 12 December 2011 15:26, K. Peachey p858snake@gmail.com wrote:
Nothing much went wrong in the planning of this feature,
Really?!
How is not having realised that this new feature would break 1000's of images and preventing it not something going wrong in the planning? (And yes, I mean "break" - they displayed correctly before and they don't now, the fact that the EXIF data was corrupt isn't anywhere near as important as how they actually display on the sites that use them.)
It was an innocent mistake and these things happen, but you need to accept that it was a mistake and consider what you can do in future to avoid such mistakes happening again.
On 13/12/11 01:36, Teofilo wrote:
Let us stop asking users to individually tag every wrong picture! Let us have some developers create a tool to find wrong pictures and rotate them back to their original orientation!
We could make a list of all images with EXIF rotation. I'm not sure how you would separate that into correctly-rotated and incorrectly-rotated images. There's not any simple way to tell whether a picture is sideways.
-- Tim Starling
On Wed, Dec 14, 2011 at 11:35:24AM +1100, Tim Starling wrote:
On 13/12/11 01:36, Teofilo wrote:
Let us stop asking users to individually tag every wrong picture! Let us have some developers create a tool to find wrong pictures and rotate them back to their original orientation!
We could make a list of all images with EXIF rotation. I'm not sure how you would separate that into correctly-rotated and incorrectly-rotated images. There's not any simple way to tell whether a picture is sideways.
I wonder... if we run/simulate the old routine vs the new routine, and we notice that there is a difference in outcome between the two, we could add a "check me" template. <scratches head>
sincerely, Kim "We Are The Robots" Bruning
On 20/12/11 12:50, Kim Bruning wrote:
On Wed, Dec 14, 2011 at 11:35:24AM +1100, Tim Starling wrote:
On 13/12/11 01:36, Teofilo wrote:
Let us stop asking users to individually tag every wrong picture! Let us have some developers create a tool to find wrong pictures and rotate them back to their original orientation!
We could make a list of all images with EXIF rotation. I'm not sure how you would separate that into correctly-rotated and incorrectly-rotated images. There's not any simple way to tell whether a picture is sideways.
I wonder... if we run/simulate the old routine vs the new routine, and we notice that there is a difference in outcome between the two, we could add a "check me" template. <scratches head>
Every image with EXIF rotation will be different between the old and the new version of MediaWiki. A Commons user (Umherirrender) already generated a list of such images, using the Toolserver database, and thousands of incorrect images were tagged for rotation.
The rate at which images were tagged was far in excess of the rate at which RotateBot was able to rotate them, so Sam Reed did some work on optimising it.
-- Tim Starling
wikimedia-l@lists.wikimedia.org