The Adobe DNG (digital negative, i.e. portable raw) format seems to pass free-content muster at a first glance.
The question then is being able to upload the things at all. How technically necessary is the 20MB limit? If we upped it to 40MB, or 100MB, would the servers melt? What are the practical issues from the view of the system administrators?
- d.
---------- Forwarded message ---------- From: Oldak Quill oldakquill@gmail.com Date: 21 Nov 2007 15:02 Subject: Re: [Commons-l] Allow Digital Negative (DNG) RAW format on Commons? (and increase filesize limit) To: andrew.gray@dunelm.org.uk, Wikimedia Commons Discussion List commons-l@lists.wikimedia.org
On 21/11/2007, Andrew Gray shimgray@gmail.com wrote:
On 21/11/2007, Brianna Laugher brianna.laugher@gmail.com wrote:
"There are indeed, some amazing images. I definitely believe that publishers could use this resource if they're in need of (one more image) to complete an existing project. But I'm uncertain about how publishable much of the content is, especially in the absence of higher resolution files (which disqualifies printing). "
So although our works are usually sufficient for web use, it seems clear that we cannot present ourselves as a serious kind of archivist, culture-recording project, without introducing a RAW format and encouraging people to use it.
Careful not to jump two steps there :-)
We mainly don't have higher resolution image files because people aren't uploading high-resolution image files to start with, not because the high-resolution JPGs or TIFFs which we have Just Aren't Good Enough(TM).
And people aren't uploading high-resolution image files because they can't. If the upload limit were increased, there are plenty of US-govt TIFFs that could be added to Commons and greatly improve our usability.
The 20MB upload limit has come up a fair few times on these mailing lists and the only objections I remember relate to whether we have the resources to handle larger files. Tell me if I'm wrong, but I can't see any problems in doubling our upload limit to 40MB straight away to enable more/better quality image and sound files to be uploaded. We should further discuss and investigate the impact of raising the limit further so we can start storing reasonable film formats and files (I, personally, cannot wait until the limit is high enough to allow us to provide reasonable quality early silent films).
Ideally, the upload limit will eventually be high enough to allow us to provide lossless data files (with classical music movements of up to ~20MB, this would need to be up to 100 MB lossless music in FLAC format). The feasibility of serving 100MB files over HTTP still needs to be discussed, but I can't see how a 40MB limit would cause us any problems or cost us significant resources.
-- Oldak Quill (oldakquill@gmail.com)
_______________________________________________ Commons-l mailing list Commons-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/commons-l
David Gerard wrote:
The 20MB upload limit has come up a fair few times on these mailing lists and the only objections I remember relate to whether we have the resources to handle larger files. Tell me if I'm wrong, but I can't see any problems in doubling our upload limit to 40MB straight away to enable more/better quality image and sound files to be uploaded. We should further discuss and investigate the impact of raising the limit further so we can start storing reasonable film formats and files (I, personally, cannot wait until the limit is high enough to allow us to provide reasonable quality early silent films).
Ideally, the upload limit will eventually be high enough to allow us to provide lossless data files (with classical music movements of up to ~20MB, this would need to be up to 100 MB lossless music in FLAC format). The feasibility of serving 100MB files over HTTP still needs to be discussed, but I can't see how a 40MB limit would cause us any problems or cost us significant resources.
I doubt our servers have any problem with normal upload usage. If newbies started uploading isos, maybe... So, based on the asumption that goodwilling users won't damage the servers (and if so it'd mean new disks are needed) nor upload copyvios which will deleted and fill space in the deleted storage: What about creating a new permission (+new group +reviewing UserRights so bureucrats can give it) 'unlimited-size-upload' to bypass the file size limit? That way, users could upload big files after requesting (a process which policy should make quite easy), as opposed to raising the limit for everyone. There could also be some group with an intermediate limit, but 11645 is a blocker (i recall it needed reviewing?) as giving it to sysops, while useful wouldn't be those most needing it, which would be some bots and certain users.
As i'm thinking about this, it could also be added as a size limit associated with each user (defaulted to $wgMaxUploadSize or a magic number meaning wiki's default) so setting it to 0 would fix the old bug 4995 and its clone 6670.
PS: I'm aware raising the limit means changing php.ini (or using url upload ;) ) . Ideally, receiving the file would be made by a CGI and not by php.
MinuteElectron wrote:
Platonides wrote:
upload ;) ) . Ideally, receiving the file would be made by a CGI and not by php.
Why? There is not much benefit (if any) of using CGI as opposed to PHP, and it would be a waste to duplicate a large ammount of code just for an installer.
MinuteElectron.
PHP reads and parses each field and then starts the script. If the ploading file is larger than the memory PHP is allowed to use for it, the input arrays become empty. A CGI acts at a lower level, so eg. you can don't need the whole file before being able to detect it's larger than what you allow (when does php check it?), you don't allow that extension or the file has a wrong extension.
My idea was a simple CGI which then passes the control to the php, not reimplementing everything. Or you could fix php. Maybe it has been improved, it's a while since i checked.
On 11/22/07, David Gerard dgerard@gmail.com wrote:
The Adobe DNG (digital negative, i.e. portable raw) format seems to pass free-content muster at a first glance.
Is this idle speculation, or do you think it would be a good idea to be uploading DNG's? What for?
Steve
On Nov 22, 2007 1:58 PM, Steve Bennett stevagewp@gmail.com wrote:
Is this idle speculation, or do you think it would be a good idea to be uploading DNG's? What for?
There's a discussion about it currently on commons-l, prompted by an inquiry from a magazine publisher about possibly using some Commons images, but unsure about their print-suitability.
For these kinds of reusers, it would be useful to allow some kind of RAW format, and DNG is probably the best one to pick, given that it's (relatively) open.
On 11/22/07, Stephen Bain stephen.bain@gmail.com wrote:
There's a discussion about it currently on commons-l, prompted by an inquiry from a magazine publisher about possibly using some Commons images, but unsure about their print-suitability.
For these kinds of reusers, it would be useful to allow some kind of RAW format, and DNG is probably the best one to pick, given that it's (relatively) open.
Interesting - even our highest quality 10mp JPEGs aren't good enough? I'm curious, because at [[WP:FPC]] we've generally considered that featured pictures should be big (heh, minimum 1000px) to allow them to be printed. It sounds like that's really nowhere near enough.
In any case, I don't think you'll see me uploading DNG's...they're huge :( And ADSL being the way it is...
Steve
On 22/11/2007, Steve Bennett stevagewp@gmail.com wrote:
On 11/22/07, Stephen Bain stephen.bain@gmail.com wrote:
There's a discussion about it currently on commons-l, prompted by an inquiry from a magazine publisher about possibly using some Commons images, but unsure about their print-suitability.
For these kinds of reusers, it would be useful to allow some kind of RAW format, and DNG is probably the best one to pick, given that it's (relatively) open.
Interesting - even our highest quality 10mp JPEGs aren't good enough? I'm curious, because at [[WP:FPC]] we've generally considered that featured pictures should be big (heh, minimum 1000px) to allow them to be printed. It sounds like that's really nowhere near enough.
This was discussed on commons-l, e.g. http://lists.wikimedia.org/pipermail/commons-l/2007-November/003178.html . DNG is the real "source" of JPGs and providing it gives reusers the greatest flexibility.
In any case, I don't think you'll see me uploading DNG's...they're huge :( And ADSL being the way it is...
Just for you, we'll allow JPGs to remain too. ;)
But anyway, back...to..the upload limit...
cheers, Brianna
On Thu, Nov 22, 2007 at 02:38:14PM +1100, Steve Bennett wrote:
On 11/22/07, Stephen Bain stephen.bain@gmail.com wrote:
There's a discussion about it currently on commons-l, prompted by an inquiry from a magazine publisher about possibly using some Commons images, but unsure about their print-suitability.
For these kinds of reusers, it would be useful to allow some kind of RAW format, and DNG is probably the best one to pick, given that it's (relatively) open.
Interesting - even our highest quality 10mp JPEGs aren't good enough? I'm curious, because at [[WP:FPC]] we've generally considered that featured pictures should be big (heh, minimum 1000px) to allow them to be printed. It sounds like that's really nowhere near enough.
I actually have an answer on this for you. :-)
For print reproduction, you typically want at least 2x the linescreen of the halftone reproduction, and maybe a little more.
For magazines, that's 120-175ls, and for books, maybe 133-200ls, depending on the market and the publication technology (bothering with screens higher than 133-150 requires that you get everything else in the publishing chain exactly right.
So, multiply those by 2 or 3, and then divide to get inches.
And remember that if your JPEGs reflect sensor oddities or sloppy compression, then you're going to scale those up as well.
So, 1000px? 4 to 5 inches, tops. If you need to fill a page, plan on needing an entire uncropped 10-12mp JPEG, and remember that fashion spreads are being shot on 22mp Hasselblads. Got $38K? :-)
Cheers, -- jra
On Nov 22, 2007 2:38 PM, Steve Bennett stevagewp@gmail.com wrote:
Interesting - even our highest quality 10mp JPEGs aren't good enough? I'm curious, because at [[WP:FPC]] we've generally considered that featured pictures should be big (heh, minimum 1000px) to allow them to be printed. It sounds like that's really nowhere near enough.
It's not just about resolution; RAW can be "developed" into different images with different settings to suit different purposes. For example, you might want to change the white balance or the colour balance to better suit print use. Of course you can alter a JPEG or something in the same way, but not without losing information.
On 22/11/2007, Steve Bennett stevagewp@gmail.com wrote:
Interesting - even our highest quality 10mp JPEGs aren't good enough?
Apparently some print people actually do say "give me the raw file! JPEG not good enough!" even if it's the same lines per inch and those of us who've had to use JPEGs in print know damn well that any JPEG artifacts will be completely invisible.
It also allows the DNGs to be developed differently for different purposes. Imagine three images uploaded at a time: the DNG, a PNG and a nicely cropped JPEG of the PNG, each for different levels of reuse.
- d.
David Gerard wrote:
The Adobe DNG (digital negative, i.e. portable raw) format seems to pass free-content muster at a first glance.
The question then is being able to upload the things at all. How technically necessary is the 20MB limit? If we upped it to 40MB, or 100MB, would the servers melt? What are the practical issues from the view of the system administrators?
Haven't we had this discussion already? Sort your mailbox by subject, it's right there.
-- Tim Starling
On 22/11/2007, Tim Starling tstarling@wikimedia.org wrote:
David Gerard wrote:
The question then is being able to upload the things at all. How technically necessary is the 20MB limit? If we upped it to 40MB, or 100MB, would the servers melt? What are the practical issues from the view of the system administrators?
Haven't we had this discussion already? Sort your mailbox by subject, it's right there.
Ah yes, sorry:
http://lists.wikimedia.org/pipermail/wikitech-l/2007-October/034344.html
So that'll be a "not yet", then.
- d.
On 23/11/2007, David Gerard dgerard@gmail.com wrote:
Ah yes, sorry:
http://lists.wikimedia.org/pipermail/wikitech-l/2007-October/034344.html
So that'll be a "not yet", then.
How do we (or rather, our developers) know when we are ready?
Of course we can't predict what will happen when we double the limit - will we get tons of stuff that is 30-40mb, or will uploads continue at more or less the same size. So - ?
cheers Brianna
On 23/11/2007, Brianna Laugher brianna.laugher@gmail.com wrote:
On 23/11/2007, David Gerard dgerard@gmail.com wrote:
Ah yes, sorry:
http://lists.wikimedia.org/pipermail/wikitech-l/2007-October/034344.html
So that'll be a "not yet", then.
How do we (or rather, our developers) know when we are ready?
Of course we can't predict what will happen when we double the limit - will we get tons of stuff that is 30-40mb, or will uploads continue at more or less the same size. So - ?
cheers Brianna
Initially we may get a bit of an increase as people upload videos that they have been unable to upload until that point (early silent films which qualify as PD-old :D). Most of our uploads will still be images, and small images at that - I really can't imagine a significant number of files in the 20-40 MB range being uploaded. Apart from early silent films, some classical music movements that are particularly long, spoken Wikipedia articles and a few high-resolution images will be uploaded. I don't see this making a big dent.
2007/11/23, Oldak Quill oldakquill@gmail.com:
Initially we may get a bit of an increase as people upload videos that they have been unable to upload until that point (early silent films which qualify as PD-old :D). Most of our uploads will still be images, and small images at that - I really can't imagine a significant number of files in the 20-40 MB range being uploaded. Apart from early silent films, some classical music movements that are particularly long, spoken Wikipedia articles and a few high-resolution images will be uploaded. I don't see this making a big dent.
Do not forgot about porn trailers ;)
AJF/WarX
wikitech-l@lists.wikimedia.org