Philip Jägenstedt wrote:

I don't suppose that the members of this list appreciate the epic Microdata vs. RDFa battle leaking into this mailing list

I wouldn't use such terms to frame the debate. The Microformats, Microdata and RDFa communities are not "battling" or working against each other - they're having a very necessary, spirited debate. Clearly, both communities are influencing the design of the other and clearly we need to have these discussions in order to make sure that we're creating the best possible technology for the future of the Web.

More importantly, the reason that all of us are working on this technology is because we care about how it is used to better humanity. At least, I hope that's why people are working on this stuff :). Certainly, we all hold Wikipedia in high regard and want what's best for this community as well.

It's not /unfortunate/ that we're having the discussion here - it was inevitable.

I'm delighted by the fact that we're even having this debate. It took ages to convince the WHAT WG that this was a problem that needed to be addressed[1] just 18 months ago.

So, we can either grit our teeth and begrudgingly go through the motions, or we can welcome the debate to come.

I choose to do the latter because I know that all of us will learn something from it and better understand the requirements for Wikimedia implementations. What we learn here will further influence guidance given to future communities, just as integrating RDFa with Drupal has influenced the advice that we may give to this community.

[ed: Microdata] is really quite intuitive and simple, with few surprises.

I agree on the first point - Microdata is pretty intuitive and simple, with few surprises. Although, I'd say the same for RDFa as well. I think we tend to forget, though, that Web semantics require a bit of effort to learn and the audience that is using the technology should be taken into account when deciding how to expose an authoring environment for the community.

I don't think that the best approach for Wikipedia is to allow direct Microdata or RDFa markup. There are already many templates in use at Wikipedia via Infobox - those templates could be leveraged to automatically generate RDFa in the same way that dbpedia.org uses those templates to generate RDF. The risk this community runs by allowing arbitrary semantic data markup is that contributors make mistakes causing half of the semantic data to be corrupted - making the rest of the data useless.

Neither Microdata nor RDFa come with few surprises for the beginner. Like all new web technologies, there is a learning curve for both of them and it's pretty similar since Microdata's design was influenced by RDFa and Microformats. More about the surprises with each, below.

[ed: Microdata] maps well to the RDF model if you want it, but doesn't force authors to think in terms of subject, predicate, object triples.

Well, Microdata /almost/ maps to the RDF model. Microdata doesn't support RDF literal typing, which is basically a fancy way of saying that you can't verify that weights, volumes, speeds, the full range of dates in different calendars, encodings such as chemical compositions, and varying other typed information is expressed cleanly by the Wikipedia contributors.

So, if you wanted to say something like this:

The speed of light is 299792458 m/s.

You would do this in RDFa:

<div about="#light"> The speed of light is <span property="measure:speed" datatype="measure:meters-per-second">299792458</span> m/s. </div>

which would generate the following triple:

<#light> measure:speed "299792458"^^measure:meters-per-second .

AFAIK, there is no way to do the equivalent in Microdata, is there Philip?

Some of you may be asking yourselves "Why is that so important?". The primary concern has to do with data validation. Good RDF vocabularies are built to be able to validate their data and this is important for large sites like Wikipedia to ensure that the data that they're exposing is valid. Since measure:speed's range is measure:meters-per-second, and meters-per-second is presumably a sub-class of xsd:decimal, then a data validator would know that it's expecting some sort of number. So, if a Wikipedia author enters some markup that generates this data:

<#baseball> measure:speed "fast enough to hurt" .

An RDF reasoner would know that not only is the data not typed, but even if it were typed, the value "fast enough to hurt" is not valid. I would expect that this most basic level of data validation would be important to Wikipedia as you want to make sure that contributors are being careful with their markup.

The above is how you would do it in RDFa. Philip, I haven't seen any work related to this in Microdata - have there been any recent developments with regard to data validation in Microdata?

...

So, we get more-or-less the same number of data items out, but there is a problem. What does "title" mean in the semantic sense? Does it mean "job title" or does it mean "work title"? The term "title" in this case is ambiguous.

No, as long as an item type is used (http://n.whatwg.org/work) there is no ambiguity. This particular item type is defined at http://www.whatwg.org/specs/web-apps/current-work/multipage/microdata.html#l...

Title here "Gives the name of the work." without ambiguity.

This is new! I'm glad this issue was addressed in Microdata as it was one of my criticisms of it when I last read the Microdata spec about six months ago. Looks like that section of the spec was last changed on October 23rd 2009? Do you know when this was put in there, Philip?

What happens when an author forgets to include itemtype? So, if somebody does this:

<div itemscope> <span itemprop="title">Emery Molyneux Terrestrial Globe</span> </div>

There's nothing to ground the "title" property. The way I'm reading the spec, it becomes ambiguous at that point, right?

RDFa is very careful to never let something like this happen... as this data ambiguity results in questionable data that you wouldn't want to pass to a reasoning agent.

Furthermore, for this particular vocabulary the mapping to RDF is defined, as such:

title: http://purl.org/dc/elements/1.1/title author: http://creativecommons.org/ns#attributionName license: http://www.w3.org/1999/xhtml/vocab#license

In other words you express the exact same information as with RDFa but without the mental overhead of triples or mixing multiple vocabularies.

... and with the added danger of expressing ambiguous data. This is not the real danger, though. While data ambiguity is really bad when it comes to data stores, centralized vocabulary management is even worse.

RDFa is built on a concept called "follow your nose", which means that all vocabulary term URLs in RDFa, such as http://purl.org/media/audio#Recording, should be dereference-able and at the end of that URL should be a machine-readable description of the vocabulary term. Preferably, a human-readable description should also exist at that URL.

Dereference http://n.whatwg.org/work and you get a 404 Error. Even worse, the Microdata work vocabulary is hard-coded in the HTML5 specification. If one wanted to extend the vocabulary, you would have to convince the only editor of that specification, who has a track record of being both very easy and very difficult to work with (based on whether or not he agrees with you), that your vocabulary term warrants addition.

There are currently 3 Microdata vocabularies in the spec[2].

To contrast, there are over 250 active RDF vocabularies[3].

That is the true power of decentralized vocabulary development, which is a corner-stone of RDFa. The RDFa community understands that Wikipedia should be in charge of choosing and extending vocabularies since this community has the appropriate domain experts. You are the experts, we are not - and it's important to recognize that in the design of any semantic data expression language.

If Wikipedia agrees that embedding semantics in their pages is of worth to humanity (and I certainly think it is of great worth), then there will come a time that this community will want to develop their own vocabulary. RDFa allows that vocabulary to be developed independently of any standards body and allows this community to have full control of it.

Sure, you could make the argument that Microdata allows RDF to be expressed (as long as you use the complete vocabulary URL), but at that point the Microdata markup is far more cumbersome than the RDFa markup. Similarly, if the goal is to express RDF, that is what RDFa was designed to accomplish.

Philip, could you give us an update on what the WHATWG sees as the publishing process for Microdata vocabularies? For example, if Wikipedia wanted to start expressing royal bloodlines using a vocabulary specific to Wikipedia, how would they go about getting that vocabulary into the HTML5 Microdata specification?

Certainly, but if wiki editors are *able* to do it by hand, then IMHO microdata is much less error-prone.

IMHO, there are ways to shoot yourself in the foot with both Microdata and RDFa - as I've outlined above. I suppose that you could use both and pick which foot you're going to shoot with which technology :), but my suggestion is that nobody should be making such generalized statements - that one is more error-prone than the other.

It's like saying that programming in Python is more error prone than programming in PHP - it depends entirely on the skill of the developer, what you're doing, and many other factors that are out of the hands of language designers.

...

However - XHTML1+RDFa is a published W3C Recommendation and it is safe

Is Wikipedia using XHTML served as application/xml+xhtml? It seems that RDFa in "XHTML" as deployed only works because consumers pretend that the data is XHTML even though it is served as text/html and treated as such by browsers. I would assume that most pages using RDFa today are neither valid XHTML, nor served with the XHTML MIME type. Any attempts to use browser DOM APIs to access the data will have surprising/confusing results, as HTML doesn't have namespaces but RDFa uses the syntax.

Frankly, this is something that nobody that uses this technology cares about because all they are ever going to see are key-value pairs (Microdata) or triples (RDFa).

This is something that only concerns browser manufacturers and RDFa parser writers. That's why there is a Microdata API, and is going to be an RDFa API. There also exist many RDFa parser implementations to abstract this low-level stuff away.

Both Microdata and RDFa are being designed to operate in "dirty" environments with invalid markup and will work regardless of the MIME type, file extension, markup botching and namespace support across websites and web browsers.

There are a number of RDFa Javascript implementations that work just[4] fine[5] on badly authored/served XHTML documents.

Besides, the Wikipedia community has done a fantastic job of generating valid XHTML:

http://validator.w3.org/check?uri=http://en.wikipedia.org/wiki/Augustus&... http://validator.w3.org/check?uri=http://en.wikipedia.org/wiki/Walyunga_Nati... http://validator.w3.org/check?uri=http://en.wikipedia.org/wiki/Nishida_Shune...

The migration to XHTML+RDFa would only require the DOCTYPE to change... which shouldn't be any more difficult than transitioning to HTML5 (or HTML5+RDFa) in the future.

Finally I will note that it is very likely that the microdata DOM APIs will get implemented in browsers, making the semantic data available to both scrapers, to native browser interfaces and to browser extensions such as user JavaScript. As an example, you might see an icon in the address bar for saving events to a calendar, or the license information of an image displayed in the native properties dialog. I stress again that I don't make any promises on behalf of Opera or any other browser vendor, these are just my predictions.

Again, this is exciting news and while I don't think Microdata is the proper solution for the Web, for the same reasons that are outlined above and many more, I'm delighted to hear that Opera is taking in-browser semantic data expression very seriously. How far we have come in just 18 months! :)

-- manu

[1]http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2008-August/015971.html [2]http://www.whatwg.org/specs/web-apps/current-work/multipage/microdata.html#m... [3]http://prefix.cc/popular/all [4]http://code.google.com/p/rdfquery/ [5]http://code.google.com/p/ubiquity-rdfa/

2010/1/16 Platonides Platonides@gmail.com:

Both of you seem to think that wikipedia editors would start placing RDF/Microdata interleaved with wiki markup. I don't think that could ever happen. The "direct markup" would be inserted into infoboxes (which are themselves wikitext, although they can get quite complex).

Something deep inside the plumbing of a template would be the place for this.

- d.

I could see the flames rising at the start of this thread, so thank you both for steering away from them.

Essentially we have a format war here, in which one or other format will win and the other will go extinct. It might be being fueled by altruism rather than capitalism, and that's brilliant, but VHS and Betamax are watching from the wings. I know sod all about either of them except what has been posted here, but I see that they're incredibly similar, but just different enough to be incompatible; and I see that they are both horribly difficult for the lay-editor to use. By that I mean that the discussion between "oh this one only requires us to put in two new attributes instead of three" misses the elephant in the room: *both* formats require us to whitelist and start filling our wikitext with the HTML tag that the most iconic piece of wikimarkup, the double brackets, have kept hidden for nine years. The reason we brought in that now-ubiquitous syntax hasn't changed: the damn thing was too difficult for the layman to understand and use.

We do, without a doubt, need to implement this metadata-capture in MediaWiki somehow, but we need to do it not only in a way that the majority of people can use and understand, but in a way which doesn't make wikitext even more complicated for everyone. If either syntax were enabled, yes it would end up at the bottom of a template stack, but a) that's not going to do anything to ensure that the tags aren't being misused elsewhere, and b) even the most careful implementation is going to manifest itself in article wikitext along the lines of ""{{person|John Smith}}, born {{birthdate|12 June 1987}}, was a {{occupation|football player}} for {{organisation|Puddlemere United}}"". Or something like that. If we encourage editors to go the whole hog on this, we might as well install SMW.

There seem to be two usecases for these systems. First, marking up the 'stuff' that MediaWiki serves: images, copyright links, author links, etc. That requires MW to be able to get hold of the raw data for, for instance, an image license; and that's begging for things like new magic words to put on the image description page, not for enabling either format directly in wikitext. The only reason to do *that*, is to support editors marking up *their own stuff*, and that's where we have problems.

I think that it would be foolish beyond belief to encourage editors to divert their volunteer time to implementing a system that could turn out to be totally anachronistic within two years; and while I think it's a laudable long-term goal I think it would thus be very silly to let editors insert *either* format directly into wikitext at this point, or for a good year to come. By far the top priority should be implementing structures by which MediaWiki can *collect* semantic data. If we implement a {{COPYRIGHT:...}} parser function, or a metadata form, or (as I've been musing over for a while) a Category-esque system that wasn't based on wikitext and so which could have a fine-grained permissions interface; we create a feature that is useful whatever happens in the metadata world. We can implement RDFa with that data, microdata, both, neither or something else entirely. We could certainly expose it through our own API. Whatever happens, editor work is not wasted.

TLDR version: jumping on either bandwagon is neither necessary nor sensible, and we should avoid getting drawn into the issue. Implementing either of the proposed methods in raw wikitext actively defeats one of the purposes of MediaWiki: to make it as easy for anyone to edit stuff. It would need to be carefully thought through, and there's no point putting that effort in until we know which format has come out on top. Adding metadata to MW's own stuff is much easier, but its groundwork should be format-independent.

in this world of economic crisis, £0.02 seems to go quite a long way :-D

--HM

On Sat, Jan 16, 2010 at 8:25 PM, Aryeh Gregor Simetrical+wikilist@gmail.com wrote:

Microdata is also safe to use for deployment.  Like other web technologies maintained by the WHATWG, it will not change once it's widely adopted, and Wikipedia adoption would probably count as wide adoption by itself.  Note that microdata, like all of HTML5, is at Last Call at the WHATWG, independent of its status as Working Draft in the W3C.

I've asked Hixie how stable Microdata is.  Since he's the sole person who decides on changes to HTML5 at the WHATWG, as you know, his answer should be authoritative.

[100116 20:35:42] <AryehGregor> I assume that if Wikipedia starts using it on a large scale and we do a MediaWiki release and such, though, you won't change it after that and break all our content, right? [100116 20:35:56] <Hixie> correct

So it's certainly stable enough for us to use.

On Sat, Jan 16, 2010 at 9:07 PM, Jesse (Pathoschild) pathoschild@gmail.com wrote:

Wikisource, especially, is in desperate need of metadata. We have some 140,000 pages on the English wiki alone that represent poems, chapters, tables of contents, and so forth. These are essentially disorganized: we have human-usable templates and categories, but there's really no good way to find works besides searching their titles.

A few years ago we combined our metadata templates into two standard templates, {{header}} (for works) and {{author}} (for authors). Every single page already provides metadata to these templates, so implementing a metadata format for machine use is trivial once it is available on MediaWiki. We *really* want this; it would allow us to index our jumbled pile of works and authors in all sorts of very useful and interesting ways. Just a few example are author search and autocompletion (we currently list works manually), finding works by genre and year and subject and so forth, searching work descriptions, and distinguishing works from subpages.

What we're talking about (microdata, RDFa, RDF, etc.) is categorically useless for Wikimedia-internal use. The only use that any of this metadata stuff has to us is exposing info to *non*-Wikimedia agents. For internal use, we can make up our own custom formats and use plain old database queries much more easily than resorting to any standard format.

For instance, we have lots of images on Commons under various licenses. *We* know which license each is under, because we use MediaWiki's category system. But *other* people (e.g., search engines) also want to know what licenses our images are under. So for this we want a standard format like microdata or RDFa, so they don't have to keep track of our internal data formats.

What Wikisource needs here is a MediaWiki extension. Standard metadata languages are not going to help at all. If no one is willing to write an extension for it now, no one will be willing with RDF support -- since that won't make the job the slightest bit easier.

Both formats have their own advantages and disadvantages. Microdata's simplicity is a significant advantage, but RDFa's built-in validation is also nice.

Neither has more built-in validation than the other. Both allow arbitrary validation. RDFa seems to allow validation to be encoded in a more machine-readable format, but whether that's an advantage at all is debatable. HTML5 does not provide a DTD, XML Schema, or any other machine-readable language description, for good reason.

On Sat, Jan 16, 2010 at 10:07 PM, Jesse (Pathoschild) pathoschild@gmail.com wrote:

Unfortunately, categories and database queries are inadequate for our needs. Someone can indeed navigate to Categories::Works::Works by genre::Non-fiction::Governmental::Biographies::Ancient biographies, and they'll find all 5 pages that someone thought to categorize to this depth. But if someone hopes to find our 1872 American biographies, they are going to be sorely disappointed.

You can do this with database queries fine -- there are already several different toolserver tools that will do category intersections for you, and a couple extensions. In fact, bog-standard search will do it for you, although AFAIK only for categories added literally (not by templates):

http://en.wikipedia.org/w/index.php?title=Special:Search&redirs=1&se...

It wouldn't be that hard to allow template-added categories too. I assume you have categories like "books published in America", "books published in 1872", and "biographies" -- if not, you can easily add them via your templates (although that wouldn't work right now with standard search AFAIK, it would work with things like CatScan).

If we simply extend MediaWiki to support metadata for works or authors, the metadata is limited to these types and fields. Public metadata can be extended and parsed in any way the local community or our content users feel useful.

Sure, but this is not internal use, so not relevant to my last post.

This is also not possible with database queries, since the metadata is not provided to the software except as part of the wiki text.

It is if you use categories. It would also be possible to hack up some tool to store all template parameter-value pairs, which are strikingly similar to the idea of RDFa triples: (article, template+parameter name, parameter value).

There is very little difference between internal and external use; it's no easier for a Wikisource editor to find those 1872 American biographies. Editors are also users.

By "internal use" I mean "use by software designed only to work with MediaWiki", not "use by Wikimedia users". Standards are only needed if we want to be useful to software that's also meant to work with other sites. That way, the software can use the same code to process both our site and the other sites, since all output the same standard markup. If the software is only processing MediaWiki sites to begin with, then standard markup is useless. (Unless it happens to expose convenient libraries, like with XML or such -- but that's probably not the case here.)

So, these metadata formats are definitely *not* useless for internal community use.

No, they really are. It's almost certainly more work for us to use a standard of any kind than to make up our own internal format, so if we only care about internal use, bothering with standards is counterproductive. The real use-cases are for external users only.

Jesse (Pathoschild <pathoschild <at> gmail.com> writes:

If we simply extend MediaWiki to support metadata for works or authors, the metadata is limited to these types and fields. Public metadata can be extended and parsed in any way the local community or our content users feel useful. Users can add their own metadata (translators? publishers? work licenses?) to templates, and add their own tools and databases to the collection.

Hi Jesse,

the use you may need seems to be a lot like what Semantic MediaWiki is offering. I don't know if Wikisource would consider it, but adding user-curated metadata using a user-generated vocabulary, and being able to query it internally (as well as exporting it externally) is pretty much what we do.

If you have any questions on it, feel free to contact me.

Cheers, denny

On Jan 17, 2010, at 16:11, Aryeh Gregor wrote:

On Sun, Jan 17, 2010 at 9:20 AM, Denny Vrandecic denny.vrandecic@kit.edu wrote:

...

the use you may need seems to be a lot like what Semantic MediaWiki is offering. I don't know if Wikisource would consider it, but adding user-curated metadata using a user-generated vocabulary, and being able to query it internally (as well as exporting it externally) is pretty much what we do.

The major problem with SMW in the past has been, AFAIK, that it's an enormous amount of code written totally separately from MediaWiki by different people, and would need to be reviewed in its entirety by someone like Tim Starling before it could be enabled on any Wikimedia site. I recall Tim looking briefly at the code and taking a few minutes to find an XSS exploit. There are also likely to be major performance issues scaling to Wikipedia (correct me if I'm wrong). So I wouldn't bet on any progress here anytime soon, especially since we're way behind on reviewing even existing core code, let alone large new extensions.

I was not talking about WIkipedia -- even though our scalability tests suggest that it could work there, but it is hard to say in advance without testing on the actual WMF server farm. I am merely talking about Wikisource, and wondering if it could be used to solve the problems they have, right now.

Furthermore, the code has had some peer review by now, it is used by sites like Wikia. Our code is getting smaller and we are incorporating comments. It would be great to get further reviews.

So, as said, I am only talking about Wikisource. I think it could be a viable solution for them.

Notably, this doesn't try to actually use the data on the wiki, so should have no scalability issues. It should also be small enough to put in core with no problems, so all MW wikis could be outputting RDF for their template parameters out of the box. My understanding is that it's expected that data providers may output RDF in whatever format is convenient to them, and someone will have to write OWL to turn this into more conventional formats. But we can output the raw data reasonably easily, at least.

Since for the requirements of Wikisource it seems that it would be helpful that the wiki itself stores and uses the data (e.g. give me all the chapters in their order of that book written by X between 1920 and 1940), I was wondering if an extension that does that could be helpful. It is obviously and entirely possible to have the metadata be generated by the RDFa-extension, the metadata be harvested by an external tool, the queries be processed by an external tool, and the result be uploaded to the wiki. It may be a bit easier for Wikisource if the wiki did it, since it could potentially enable more users to perform these tasks.

In the case of Wikisource I'd further suggest to switch off the additional annotation syntax of SMW on go for a modus were the templates do the whole annotation, but that again is an implementation detail that has to be decided by the Wikisource community.

Cheers, denny

Aryeh Gregor Simetrical+wikilist@gmail.com wrote:

[...]

...

Also, it's a bit sad that so many toolserver tools are standalone and are not a part of MediaWiki distribution. That tool should be a part of Special:Search.

Most toolserver tool authors just don't bother applying for commit access for whatever reason. Most tools also either perform badly and/or would need to be rewritten to meet coding standards. Toolserver roots routinely have to kill processes for using up unreasonable amounts of resources. [...]

How many of those tools would stand a chance as extensions of not being disabled in $wgMiserMode? If such a small fea- ture as the namespace filter in Special:Linksearch risks server meltdown (bug #10593), I doubt more complex searches are on the horizon.

Tim

On 18/01/10 14:46, Daniel Kinzler wrote:

...

The point is, anyway, that enabling something like SMW (probably with fewer features) is orthogonal to RDFa/microdata/RDF support -- the extension could incidentally output RDF or whatnot, but it doesn't matter for internal use.

Perhaps the right approach for us would be to have "some" syntax for providing this info, and then generating html5 microdata and/or rdfa into the rendered html, write the triple into a smw backend store, and provide rdf/xml/n3/whatever output via the api.

there are three aspects here: specify, store, output. perhaps we should look at them separately.

-- daniel

I definitely wouldn't recommend a flat triples store as the only storage representation.

Based on past experience with just such a system, while it's formally semantically equivalent to higher-level descriptions, it's definitely much harder to munge, because you have to reverse-engineer all the reification that was needed to flatten the data into triples in order to be able to see the higher-level patterns; it's much easier to just store the higher-level description in the obvious natural way, and generate the triples representation, and any other metadata output needed, from that.

-- Neil

Before we get into this thread too deeply, for those that are not familiar with semantic data, RDF, RDFa or why any of this stuff applies to Wikipedia, there are two very short videos that explain the concepts at a high-level (apologies, as they're a bit dated):

Intro to the Semantic Web (6 minutes) http://www.youtube.com/watch?v=OGg8A2zfWKg

RDFa Basics (9 minutes) http://www.youtube.com/watch?v=ldl0m-5zLz4

Aryeh Gregor wrote:

What we're talking about (microdata, RDFa, RDF, etc.) is categorically useless for Wikimedia-internal use.

Not necessarily. Javascript can use the RDFa on the page to generate more intuitive interfaces for the page. To give an example - we use the RDFa expressed in our music pages:

http://bitmunk.com/media/6995806

to drive the music player application via Javascript - by parsing the RDFa and feeding the sample URLs to the player.

To give a less than ideal example - Wikipedia could use data on the page to provide interactive discovery of concepts expressed on the page (such as automatically fetching and parsing RDFa on a related page to display more factual information on the current page). The gist of what I'm getting at is to not dismiss the value of having a standardized mechanism for embedded page data - you get to use it internally and externally. The more data you expose, the greater the possibility of somebody figuring out how to use the data in amazing new ways.

Aryeh Gregor wrote:

I'll emphasize from the start that I do *not* think either RDFa or microdata is suitable for dbpedia.org-style content. There's no reason we should put that in the HTML output, where it will take up tons of space and not be useful to HTML consumers (e.g., browsers and search engines).

Placing this data in your HTML documents has a direct impact on browsers and search engines. Browsers can collect triples and use them later to help you answer questions that you may have about a particular subject. Search engines can crawl the HTML and make their indexes more accurate based on semantic data that Wikipedia's pages expose.

RDF/XML, which was largely unsuccessful, was designed to be used for publishing in a dual-stream setup. It was expected that web publishers would publish semantic data beside web page data, just as you've proposed that Wikipedia does, but this proved to be far too difficult for most sites to manage both types of serializations.

Wikipedia is already short on developers, creating a new data stream is just going to exacerbate the problem. Besides, the way Wikipedia seems to be capturing data is via wikitext, not direct database entries. In effect, this community's database exists in the wikitext.

Aryeh Gregor wrote:

On the other other other hand, RDFa 1.1 is under development and looks like it will make major changes, so from that perspective microdata is arguably more stable.

There are new features going into RDFa 1.1, but classifying them as "major" changes makes it sound like RDFa 1.1 isn't going to be backwards-compatible with RDFa 1.0, when it most definitely is going to be backwards-compatible (except possibly for XMLLiterals, which was our bad).

The statement that "Microdata" is more stable because there are new features going into RDFa 1.1 is illogical. For example: just because there are new features going into the next version of Apache doesn't mean that it's any less "stable" for those that are using the current version today.

Aryeh Gregor wrote:

So, it's complicated. :) But from our perspective, I don't think there's a big difference in terms of stability or standard-ness, so I skipped over all this.

There's a huge difference in both stability and standard-ness - XHTML+RDFa is a W3C REC - it's a standard. Microdata and HTML+RDFa aren't even close to becoming a W3C REC. That's very important information for this community to consider.

When do you think that Microdata is going to be a REC at the W3C?

There were changes to the Microdata spec made by Ian less than 12 hours ago (January 18th 2010). If a spec is being actively edited, I don't think it's a good idea to say that it's stable and ready for deployment:

http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2010-January/024760.html

You are skipping over some pretty important stuff, kemosabe. :)

Aryeh Gregor wrote:

so converting the microdata graph to RDFa might be easier than the reverse.

Microdata's underlying model is triples as well - Microdata allows the limited expression of RDF. Since RDFa also supports the expression of RDF more formally, you can map Microdata to RDFa easier than you can map RDFa to Microdata (for some value of "easier).

You cannot, however, express RDF fully in Microdata - it is impossible in cases where it matters to Wikipedia (like data-typing).

Microdata doesn't support data typing (via @datatype), data value overriding (via @content), doesn't support URI short-handing via CURIEs (via @xmlns:PREFIX), and it doesn't support anonymous subjects via bnodes (blank nodes). The @datatype thing, @content thing and CURIE thing affects Wikipedia, not supporting bnodes doesn't necessarily impact the Wikipedia community, AFAICT.

Aryeh Gregor wrote:

I also think microdata is much easier to author for people with an HTML (not RDF) background -- template editors tend to have a good working knowledge of HTML, but not web-data technologies. I'd be interested in what Manu (or other RDFa supporters) has to say here.

I do think that Microdata has that going for it - in that the property names such as @itemref, @itemprop, etc. are easier to understand that @about, @datatype, @rel/@rev, and @content.

I'm all for making it easier for web authors to write this stuff, so the consistency of the itemXYZ attributes in Microdata was a good move. We didn't choose to do that for RDFa because we wanted to make the mapping from HTML to RDF explicit. The down-side with that is it requires authors to either have their RDFa autogenerated for them (which is the best thing for RDFa and Microdata), or it requires them to sit through a 10 minute tutorial on RDF (like the video at the top of this e-mail).

I do also think that Microdata has made several really big mistakes that we made in the Microformats community that were corrected in the RDFa community. Namely, not using CURIEs and adding the requirement that all URLs are repeated as many times as they're used. It's fine as an option, but not that great if one has to repeat a URL 50 times in a web page... which Wikipedia will eventually have to do if it is using Microdata.

http://rdfa.info/wiki/Developer-faq#Authoring

The FAQ above, which is a work in progress, is a good introduction to some of the common criticisms against RDFa and the reasoning behind the design decisions, for those that are interested.

The FAQ also addresses the fallacy that RDFa markup is, for real-world data, more verbose than Microdata markup.

Aryeh Gregor wrote:

Neither has more built-in validation than the other. Both allow arbitrary validation. RDFa seems to allow validation to be encoded in a more machine-readable format, but whether that's an advantage at all is debatable.

That's provably false. Microdata vocabulary validation is hard-coded in the specification. Dan Brickly and Ian Hickson had an IRC conversation about just this today[1]. In order to validate Microdata, you must first either convert it to RDF and even if you do, it will fail attempts to validate the literals that should have a datatype. If you want a Microdata vocabulary validator, you have to create one for each vocabulary... just like we had to do in the Microformats community, which some of us now recognize as a catastrophic mistake.

RDFa, via RDF, allows arbitrary data validation - one validator with any number of vocabularies. Microdata does not allow arbitrary validation - there must be one hard-coded validator per vocabulary.

-- manu

[1]http://krijnhoetmer.nl/irc-logs/whatwg/20100118#l-219

On Mon, Jan 18, 2010 at 23:34, Manu Sporny msporny@digitalbazaar.com wrote:

You cannot, however, express RDF fully in Microdata - it is impossible in cases where it matters to Wikipedia (like data-typing).

I'm not a Wikipedia developer or particularly active editor, but it sounds quite doubtful that XML Schema Datatypes matters to Wikipedia. Perhaps I haven't understood RDFa, but surely the vocabulary must define the datatype? If not, is @datatype a mandatory attribute that just adds dead weight all over the place? And if vocabularies do define the datatypes, why do you need to override them?

I do also think that Microdata has made several really big mistakes that we made in the Microformats community that were corrected in the RDFa community. Namely, not using CURIEs and adding the requirement that all URLs are repeated as many times as they're used. It's fine as an option, but not that great if one has to repeat a URL 50 times in a web page... which Wikipedia will eventually have to do if it is using Microdata.

There are other solutions to the "URLs are long" problem than prefix schemes. Incidentally http://n.whatwg.org/work is rather short, and I hope future vocabularies will have the good taste to use even shorter URLs.

That's provably false. Microdata vocabulary validation is hard-coded in the specification. Dan Brickly and Ian Hickson had an IRC conversation about just this today[1]. In order to validate Microdata, you must first either convert it to RDF and even if you do, it will fail attempts to validate the literals that should have a datatype.

Is the only kind of validation that RDF provides validation that something is the same kind of data it claims to be? That sounds similar to and as unhelpful as doctypes. What if the author doesn't set the datatype?

If you want a Microdata vocabulary validator, you have to create one for each vocabulary... just like we had to do in the Microformats community, which some of us now recognize as a catastrophic mistake.

RDFa, via RDF, allows arbitrary data validation - one validator with any number of vocabularies. Microdata does not allow arbitrary validation - there must be one hard-coded validator per vocabulary.

What are the exact mechanisms here? Does a RDFa validator dereference all predicates and try to get an RDF Schema to validate against? Doesn't that destroy any web server which hosts schemas for popular vocabularies (like with W3C doctypes)? On the other hand, if only the document itself is used, what kind of validation can be meaningful?

In any case, validators for microdata is something to be worked on, but I don't think either dereferencing vocabulary URLs or an official schema language is likely to be part of the solution (the latter because you need a full programming language to validate certain types of data, not just grammar rules).