I doubt that our licenses and Google's license filter are playing together well. I am regularly pinged by a Google guy who is desperate to get Commons data into Google Image Search in a more systematic way. If any of you think this project is interesting I can totally get you off the ground. It will not be hard. On 10/13/11 12:24 PM, Federico Leva (Nemo) wrote: -- Neil Kandalgaonkar (| <neilk(a)wikimedia.org>

Google guys inquiring about Commons? That’s cool to know. But I don’t recall reading anywhere on Commons about Google folks pinging the WMF about Commons indexing. Did I miss it, or it was never announced? If the latter, how volunteer Commons hackers can possibly give a hand if they’re not aware of it? I totally understand the WMF/devs have other stuff to do/priorities. But I think such info should be communicated to communities. Especially if they can help as « it will not be hard ». No? -- Jean-Frédéric * (cc-ing Sumana)*

On 10/13/11 1:35 PM, Rayson Ho wrote: 1) Figure out some scheme whereby the actual license is available in the database, not merely expressed in human-readable HTML. This is hard, so it's where I gave up. Timo (Krinkle) and Roan Kattouw were working on this for a bit but they were pulled off to do other things. To do this right we'd create a new namespace like License: and then connect that to some database entity. We could then connect those to existing license templates. 2) Once that's done, from a daily cronjob or something, generate Sitemaps (a summary of all our content) compatible with the Google Image Search extended syntax. http://www.google.com/support/webmasters/bin/answer.py?answer=178636 A less organized dump from my brain here: http://www.mediawiki.org/wiki/User:NeilK/Sitemaps Google *can* do any number of things. However, they will probably not do any custom development work for Commons. By 2011 standards, Commons is a relatively small image repository. Flickr has billions of images, and it's not even the most popular photo host. Facebook, although inaccessible to Google, adds several billion images to its repository *per week*. Commons may have some of the "best of the web" images for illustration purposes, so it is a high-value thing for Google to crawl. So yeah, it's enough for them to assign a guy to talk to me every few months or so. But not enough that they will assign developers. They wouldn't have even bothered pinging us if that were the case. Commons has no real way to communicate licenses to Google. Templates create human-readable HTML, not machine-parseable legal information. If someone edited the CC master template tomorrow to look a bit prettier, anything that was trying to parse licenses from HTML would break. Google has a standard for us to tell them the license, in the extended Sitemap syntax for images, linked to above. That's what we should do, because it would make that information available to Google, and potentially to any other search engines that can read that standard. Hell yes! -- Neil Kandalgaonkar (| <neilk(a)wikimedia.org>

On 13 October 2011 21:58, Neil Kandalgaonkar <neilk(a)wikimedia.org> wrote: Do they read microformats/RDF? Adding those to the templates wouldn't be unfeasible. - d.

Is this something that could be solved implementing OAI-MPH protocol in Commons? I don't know how hard it would be, but it seems that exporting metadata (as a License is) is becoming more and more important... Last time we spoke with a dev about it (Berlin Hackathon), he basically said: "We need to change mediawiki..." Aubrey 2011/10/14 ChrisiPK <chrisipk(a)gmail.com>

I didn't ask, but perhaps I could find out. It's an interesting idea, although to some degree it's once again postponing the necessary IMO work of putting licenses in the database. I mean, Commons regards correct licensing as one of the most important activities, and yet licenses aren't a real object in the system. It's very difficult to gather even basic information about how licenses are used on Commons. Anyway as far as I can tell, microformats are dead. However, HTML5 microdata is on its way. http://www.w3.org/TR/microdata/ A Google employee wrote that spec, but that's not a guarantee it will actually work with anything, or that Google Image Search has any idea he wrote it. ;) On 10/13/11 2:02 PM, David Gerard wrote: -- Neil Kandalgaonkar (| <neilk(a)wikimedia.org>

On 14 October 2011 09:10, Neil Kandalgaonkar <neilk(a)wikimedia.org> wrote: Eurgh. We don't really want to get into the RDFa/microformats/microdata holy wars... :-) Frankly, given the movement's mission and commitments, using RDFa seems most sensible. Though I agree with Neil that having licences as a first-level object would be nice-to-have, the templates give us the ability to achieve proper machine-readable licence-tagging right now (well, technically they'd apply to the page not the image, but Google could easily code around that, or in a pinch we could have an extension that extracted the RDFa attributes and applied them to the IMG and A elements). Yours, -- James D. Forrester jdforrester(a)wikimedia.org | jdforrester(a)gmail.com [[Wikipedia:User:Jdforrester|James F.]]

On 10/13/2011 5:02 PM, David Gerard wrote: I'm going to say that this is B.S. It seems everybody in business thinks it's easy to write GUI applications (where you really spend four months rewriting the requirements again and again and doing testing that never ends) and hard to write screen scrapers. (where you sometimes get it to work in four minutes) I built a rather complicated system that reads the Wiki markup and extracts a whole bunch of metadata. This system was fairly accurate but eventually reached a plateau of what it could do. It had trouble extracting licenses all the time because templates are wrapped up inside of templates which are wrapped up inside of templates and so on. The old system often had to deal with contradictory data -- for instance, there's a certain guy who uses {self} templates on photos that came from Flickr. Nobody really noticed that there's a problem here because the HTML markup looks superficially O.K. The issue is that HTML output on Commons is tested every day, and the ability to get semantics out of the inner markup doesn't get tested. "Fifth wheel" features (microformats, etc.) are even more likely to break without being noticed since nobody actually uses them... Later on I developed a much simpler heuristic: extract all hyperlinks from the HTML and filter for links that point to licenses. For instance, http://commons.wikimedia.org/wiki/File:2011-03-09-fort-du-lomont-10.jpg has a link to http://creativecommons.org/licenses/by/3.0/deed.en This is as easy to read as any kind of structured metadata could ever be. And it's not a "fifth wheel", it's actually visible in the HTML markup, so if it's wrong people will notice.

On 10/14/11 1:21 AM, John Vandenberg wrote: We actually provide an OAI-PMH feed of new image uploads already, but without having the licensing and attribution information in the database, it is of limited use. Ryan Kaldari

On 10/14/11 10:46 AM, Federico Leva (Nemo) wrote: Yep, that's it. It has some fairly expensive DB queries, so access is restricted to major search engine vendors as far as I understand. I'm not really the right person to ask about it though. Ryan Kaldari

2011/10/14 Ryan Kaldari <rkaldari(a)wikimedia.org> Ryan, who is the right person to ask? Everyone who is interested in having structured metadata for Commons always stumble upon that mysterious page (at leats, the few who are insterested in OAI-MPH), and I think it would be good to understand more of it once for good. At least, we could know if could be useful or not. As you perfectly know, the issue of not having a reliable/proper system for "metadata communication" (call is as you like) is a perennial issue, and I bet many of us would like to have it on developers priorities (at least, for Commons) :-) Thanks for the information! Aubrey Ryan Kaldari