Message: 7 Date: Thu, 1 Dec 2011 12:36:02 -0500 From: Chad innocentkiller@gmail.com Subject: Re: [Wikitech-l] Proposal for new table image_metadata To: Wikimedia developers wikitech-l@lists.wikimedia.org Message-ID: CADn73rNuSX8RegdUBCeSYG8Mz1qg5SA49VAmB5eD_Y-vB-L4dw@mail.gmail.com Content-Type: text/plain; charset=UTF-8
On Thu, Dec 1, 2011 at 12:34 PM, William Lee wlee@wikia-inc.com wrote:
I'm a developer at Wikia. We have a use case for searching through a file's metadata. This task is challenging now, because the field Image.img_metadata is a blob.
We propose expanding the metadata field into a new table. We propose the name image_metadata. It will have three columns: img_name, attribute (varchar) and value (varchar). It can be joined with Image on img_name.
On the application side, LocalFile's load* and decodeRow methods will have to be changed to support the new table.
One issue to consider is the file archive. Should we replicate the metadata table for file archive? Or serialize the data and store it in a new table (something like fa_metadata)?
Please let us know if you see any issues with this plan. We hope that this will be useful to the MediaWiki project, and a candidate to merge back.
That was part of bawolff's plan last summer for GSoC when he overhauled our metadata support. He got a lot of his project done, but never quite got to this point. Something we'd definitely like to see though!
-Chad
Chad beat me to writing essentially what I was going to say. Basically my project ended up being more about extracting more information, and i didn't really touch what we did with it after we extracted.
However, it should be noted that storing the image metadata nicely is a little more complicated then it appears at first glance (and that's mostly my fault due to stuff i added during gsoc ;)
Basically there's 4 different types of metadata values we store (in terms of the types of metadata you think of when you think EXIF et al. We stuff other stuff into img_metadata for extra fun) *Normal values - Things like Shutter speed = 1/110 *unordered array - For example we can extract a "tags" field that's an arbitrary list of tags, The subject field (from XMP) is an unordered list, etc *Ordered array - Not used for a whole lot Most prominent example is the XMP author field is supposed to be an ordered list of authors, in order of importance. Honestly, we could just ditch caring about this, and probably nobody would notice. *Language array - XMP and PNG text chunks support a special value where you can specify language alternatives. In essence this looks like an associative array of "lang-code" => "translation of field into that lang", plus a special fallback "x-default" dummy lang code. *Also Contact info and software fields are stored kind of weirdly....
Thus, just storing a table of key/value pairs is kind of problematic - how do you store an "array" value. Additionally you have to consider finding info. You probably want to efficiently be able to search through lang values in a specific language, or for a specific property and not caring for the language.
Also consider how big a metadata field can get. Theoretically it's not really limited, well I don't expect it to be huge, > 255 bytes of utf-8 seems a totally reasonable size for a value of a metadata field.
Last of all, you have to keep in mind all sorts of stuff is stored in the img_metadata. This includes things like the text layer of Djvu files (although arguably that shouldn't be stored there...) and other handler specific things (OggHandler stores some very complex structures in img_metadata). Of course, we could just keep the img_metadata blob there, and simply stop using it for "exif-like" data, but continue using it for handler specific ugly metadata that's generally invisible to user [probably a good idea. The two types of data are actually quite different].
One issue to consider is the file archive. Should we replicate the metadata table for file archive? Or serialize the data and store it in a new table (something like fa_metadata)?
Honestly, I wouldn't worry about that, especially in the beginning. As far as i know, the only place fa_metadata/oi_metadata is used, is that you can request it via api (I suppose it's copied over during file reverts as well). I don't think anyone uses that field on archived images really. (maybe one day bug 26741 will be fixed and this would be less of a concern).
Anyhow, I do believe it would be awesome to store this data better. I can definitely think of many uses for being able to efficiently query it. (While I'm on the subject, making lucene index it would also be awesome).
Cheers, Bawolff
p.s. If its helpful - some of my ideas from last year for making a new metadata table are at http://www.mediawiki.org/wiki/User:Bawolff/metadata_table and the thread http://thread.gmane.org/gmane.science.linguistics.wikipedia.technical/48268 . However, they're probably over-complicated/otherwise not ideal (I was naive back then ;). They also try and be able to encode anything encodable by XMP, which is most definitely a bad idea, since XMP is very complicated...
On Thu, Dec 1, 2011 at 8:49 PM, bawolff bawolff+wn@gmail.com wrote:
Thus, just storing a table of key/value pairs is kind of problematic - how do you store an "array" value. Additionally you have to consider finding info. You probably want to efficiently be able to search through lang values in a specific language, or for a specific property and not caring for the language.
Two easiest things based on my previous experience: 1) separate values with \x00, making them easy to split after extracting a row 2) store multiple entries with an index field, making it easy to query for potentially multiples
Also consider how big a metadata field can get. Theoretically it's not really limited, well I don't expect it to be huge, > 255 bytes of utf-8 seems a totally reasonable size for a value of a metadata field.
Last of all, you have to keep in mind all sorts of stuff is stored in the img_metadata. This includes things like the text layer of Djvu files (although arguably that shouldn't be stored there...) and other handler specific things (OggHandler stores some very complex structures in img_metadata). Of course, we could just keep the img_metadata blob there, and simply stop using it for "exif-like" data, but continue using it for handler specific ugly metadata that's generally invisible to user [probably a good idea. The two types of data are actually quite different].
On text: DjVu and PDF files can optionally contain flattened searchable text, which we extract so it can be used for things like Extension:ProofreadPage and, potentially, search indexing:
https://bugzilla.wikimedia.org/showdependencytree.cgi?id=21061&hide_reso...
Currently this gets stuffed into the metadata blob along with the exif data etc, and can make metadata blobs *very* large if there are hundreds of pages of text.
If extracted page text is stored in a better key-value store, we should make sure it doesn't get pulled in to backwards-compatible metadata blobs (if we keep em around as they are now) -- but they should be accessible through some API.
One issue to consider is the file archive. Should we replicate the metadata
table for file archive? Or serialize the data and store it in a new
table
(something like fa_metadata)?
Honestly, I wouldn't worry about that, especially in the beginning. As far as i know, the only place fa_metadata/oi_metadata is used, is that you can request it via api (I suppose it's copied over during file reverts as well). I don't think anyone uses that field on archived images really. (maybe one day bug 26741 will be fixed and this would be less of a concern).
That reminds me: ForeignAPIRepo (InstantCommons) wants to be able to transfer the metadata at least for current versions; API formats should remain compatible if possible in order for data to continue to be transferred to clients running old versions.
-- brion
On 12/05/2011 08:07 PM, Brion Vibber wrote:
If extracted page text is stored in a better key-value store, we should make sure it doesn't get pulled in to backwards-compatible metadata blobs (if we keep em around as they are now) -- but they should be accessible through some API.
One thing to consider is what happens if a user edits metadata, e.g. adds EXIF data that was lost by cropping, or if a new (cropped) version of the image is uploaded with the same name.
Another thing is image annotations, that are today always added as plain text in the image description page, http://commons.wikimedia.org/wiki/Commons:Image_annotations
A third thing is timed text (video subtitles), which today is added in separate subpages, one for each language, http://commons.wikimedia.org/wiki/Commons:Timed_Text
A fourth thing is proofreading: If OCR text was extracted from a PDF or DJvu and then proofread in Wikisource, shouldn't the next person that downloads the PDF file get the new text?
Perhaps a system for managing image + text, including wiki editing, could address all four things above? In particular, image annotations and OCR text are both tied to coordinates in the image. (And timed text is tied to a time position in a video stream.) So why are they separate systems?
Thanks to everyone for your feedback about this plan.
After careful consideration, we have decided to discontinue our plan. It does not go far enough to support the XMP standard. Instead, we will use the field Image.img_metadata for the time being.
William
On Thu, Dec 1, 2011 at 8:49 PM, bawolff bawolff+wn@gmail.com wrote:
Message: 7 Date: Thu, 1 Dec 2011 12:36:02 -0500 From: Chad innocentkiller@gmail.com Subject: Re: [Wikitech-l] Proposal for new table image_metadata To: Wikimedia developers wikitech-l@lists.wikimedia.org Message-ID: <
CADn73rNuSX8RegdUBCeSYG8Mz1qg5SA49VAmB5eD_Y-vB-L4dw@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
On Thu, Dec 1, 2011 at 12:34 PM, William Lee wlee@wikia-inc.com wrote:
I'm a developer at Wikia. We have a use case for searching through a
file's
metadata. This task is challenging now, because the field Image.img_metadata is a blob.
We propose expanding the metadata field into a new table. We propose
the
name image_metadata. It will have three columns: img_name, attribute (varchar) and value (varchar). It can be joined with Image on img_name.
On the application side, LocalFile's load* and decodeRow methods will
have
to be changed to support the new table.
One issue to consider is the file archive. Should we replicate the
metadata
table for file archive? Or serialize the data and store it in a new
table
(something like fa_metadata)?
Please let us know if you see any issues with this plan. We hope that
this
will be useful to the MediaWiki project, and a candidate to merge back.
That was part of bawolff's plan last summer for GSoC when he overhauled our metadata support. He got a lot of his project done, but never quite
got
to this point. Something we'd definitely like to see though!
-Chad
Chad beat me to writing essentially what I was going to say. Basically my project ended up being more about extracting more information, and i didn't really touch what we did with it after we extracted.
However, it should be noted that storing the image metadata nicely is a little more complicated then it appears at first glance (and that's mostly my fault due to stuff i added during gsoc ;)
Basically there's 4 different types of metadata values we store (in terms of the types of metadata you think of when you think EXIF et al. We stuff other stuff into img_metadata for extra fun) *Normal values - Things like Shutter speed = 1/110 *unordered array - For example we can extract a "tags" field that's an arbitrary list of tags, The subject field (from XMP) is an unordered list, etc *Ordered array - Not used for a whole lot Most prominent example is the XMP author field is supposed to be an ordered list of authors, in order of importance. Honestly, we could just ditch caring about this, and probably nobody would notice. *Language array - XMP and PNG text chunks support a special value where you can specify language alternatives. In essence this looks like an associative array of "lang-code" => "translation of field into that lang", plus a special fallback "x-default" dummy lang code. *Also Contact info and software fields are stored kind of weirdly....
Thus, just storing a table of key/value pairs is kind of problematic - how do you store an "array" value. Additionally you have to consider finding info. You probably want to efficiently be able to search through lang values in a specific language, or for a specific property and not caring for the language.
Also consider how big a metadata field can get. Theoretically it's not really limited, well I don't expect it to be huge, > 255 bytes of utf-8 seems a totally reasonable size for a value of a metadata field.
Last of all, you have to keep in mind all sorts of stuff is stored in the img_metadata. This includes things like the text layer of Djvu files (although arguably that shouldn't be stored there...) and other handler specific things (OggHandler stores some very complex structures in img_metadata). Of course, we could just keep the img_metadata blob there, and simply stop using it for "exif-like" data, but continue using it for handler specific ugly metadata that's generally invisible to user [probably a good idea. The two types of data are actually quite different].
One issue to consider is the file archive. Should we replicate the
metadata
table for file archive? Or serialize the data and store it in a new
table
(something like fa_metadata)?
Honestly, I wouldn't worry about that, especially in the beginning. As far as i know, the only place fa_metadata/oi_metadata is used, is that you can request it via api (I suppose it's copied over during file reverts as well). I don't think anyone uses that field on archived images really. (maybe one day bug 26741 will be fixed and this would be less of a concern).
Anyhow, I do believe it would be awesome to store this data better. I can definitely think of many uses for being able to efficiently query it. (While I'm on the subject, making lucene index it would also be awesome).
Cheers, Bawolff
p.s. If its helpful - some of my ideas from last year for making a new metadata table are at http://www.mediawiki.org/wiki/User:Bawolff/metadata_table and the thread http://thread.gmane.org/gmane.science.linguistics.wikipedia.technical/48268 . However, they're probably over-complicated/otherwise not ideal (I was naive back then ;). They also try and be able to encode anything encodable by XMP, which is most definitely a bad idea, since XMP is very complicated...
wikitech-l@lists.wikimedia.org