Roan Kattouw wrote: I agree. It should also be fixed. If we wanted to map it to a page/revision format, it seems quite straightforward. I'm missing something, right?

On Fri, Jan 21, 2011 at 10:43 AM, Roan Kattouw <roan.kattouw(a)gmail.com>wrote;wrote: If one's clever about it, this could probably actually be done on-the-fly in a reasonably non-evil fashion. Image version data isn't used as widely as revisions; eg things like Special:Contributions always needed direct access to old revs looked up by author, whereas I think image old versions are pretty much only pulled up by title, via the image record. There are also relatively few revisions per file -- old images usually only have a few revisions, and cases of thousands of versions are I suspect very rare -- which would make the actual conversion work relatively lightweight for each file record. Further optimizing by delaying on-demand migration of a record until write time could also keep it from being a sudden database & i/o sink. If indirect lookups won't be needed, we can just keep reading the existing image/oldimage records until they need to be updated on modification (or get updated by a background task at leisure). -- brion

2011/1/21 Platonides <Platonides(a)gmail.com>om>: We haven't thought about it in detail. But it would be a fairly large change and require changes throughout the software, as well as possibly elsewhere in the schema. Sure, it can be dealt with. It's just that it'd be an epic upgrade :) Roan Kattouw (Catrope)

On 01/20/2011 05:00 PM, Platonides wrote: I would also say its probably best to pass metadata from the wikitext to the tables via a parser function. Similar to categories, and all other "user edited" metadata. This has the disadvantage that its not easy 'as easy' to edit via structured api entry point, but has the advantage of working well with all the existing tools, templates and versioning. --michael

The interest of wikisource project for a formal and standardyzed set of book metadata (I presume from Dublin Core) into a database table is obviuos. Some preliminary tests into it.source suggest that templates and Labeled Section Transclusion extension could have a role as "existing wikitext conteiners for semantized variables"; the latter perhaps more interesting than the former one, since their content can be accessed directly from any page I'd like that book metadata would be considered from the beginning of this interesting project. Alex

On Fri, Jan 21, 2011 at 3:36 AM, Michael Dale <mdale(a)wikimedia.org> wrote: This is actually the biggest decision that has been made, the rest is mostly implementation details. (Please note that I'm not presenting you with a fait accompli, it is of course still possible to change this) Handling metadata separately from wikitext provides two main advantages: it is much more user friendly, and it allows us to properly validate and parse data. Having a clear separate input text field "Author: ____" is much more user friendly {{#fileauthor:}}, which is so to say, a type of obscure MediaWiki jargon. I know that we could probably hide it behind a template, but that is still not as friendly as a separate field. I keep on hearing that especially for newbies, a big blob of wikitext is plain scary. We regulars may be able to quickly parse the structure in {{Information}}, but for newbies this is certainly not so clear. We actually see that from the community there is a demand for separating the meta data from the wikitext -- this is after all why they implemented the uselang= hacked upload form with a separate text box for every meta field. Also, a separate field allows MediaWiki to understand what a certain input really means. {{#fileauthor:[[User:Bryan]]}} means nothing to MediaWiki or re-users, but "Author: Bryan___ [checkbox] This is a Commons username" can be parsed by MediaWiki to mean something. It also allows us to mass change for example the author. If I want to change my attribution from "Bryan" to "Bryan Tong Minh", I would need to edit the wikitext of every single upload, whereas in the new system I go to Special:AuthorManager and change the attribution. Categories is a good example of why metadata does not belong in the wikitext. If you have ever tried renaming a category... you need to edit every page in the category and rename it in the wikitext. Commons is running multiple bots to handle category rename requests. All these advantage outweigh the pain of migration (which could presumably be handled by bots) in my opinion. Best regards, Bryan

On Jan 22, 2011 at 21:04 Platonides wrote: So PHP would extract {{#author:4}} and {{#license:12}} from the textblob when showing the editpage. And show the remaining wikitext in the <textarea> and the author/ license as seperate form elements. And upon saving, generate "{{#author:4}} {{#license:12}}\n" again and prepend to the textblob. Double instances of these would be ignored (ie. stripped automatically since they're not re-inserted to the textblob upon saving). One small downside would be that if someone would edit the textarea manually to do stuff with author and license, the next edit would re-arrange them since they're extracted and re-insterted thus showing messy diffs. (not a major point as long as it's done independant from JavaScript, which it can be if done from core / php). If that's what you meant, I think it is an interesting concept that should not be ignored, however personally I am not yet convinced this is the way to go. But when looking at the complete picture of up/down sides, this could be something to consider. -- Krinkle

On Sat, Jan 22, 2011 at 10:09 PM, Platonides &lt;Platonides(a)gmail.com&gt; wrote: OK, my 2 cents: I would be in favour of extracting data from the {{Information}} template via the parser, but we talked about this over a year ago at the Paris meeting, and it was deemed too complicated (black caching magick etc.), and noone has stepped forward to do anything along those line, so I guess it's dead and buried. Things like {{#author:4}} seem to be a nice hack to Get Things Done (TM). As was mentioned before, the temptation is great to expand it into a generic triplet storage a la Semantic MediaWiki, but that would probably complicate things to an extend where nothing gets done, again. But one thing comes to mind: If someone implements an abstraction layer ("4" to a specific author) anyway, it should be dead simple to use it for tags as well. Just allow multiple {{#tag}}s per page (as opposed to {{#author}}), done. The same code that will allow for editing author and license information centrally should make it possible to edit tag information, i18n for example, so the tag display could be in the current user language (with "en" fallback). Search for tags i18n-style could be possible as well, if the translation information is encoded machine-readable as well, e.g. as language links ([[de:Pferd]] on the [[Tag:Horse]] page). It might be too much to try to activate all of that in the first round, but IMHO the code should keep the use as tags in mind; it would be dreadful to waste such an opportunity. Cheers, Magnus

"Platonides" &lt;Platonides(a)gmail.com&gt; wrote in message news:ihfd31$buv$1@dough.gmane.org... Eeewwwwww.... What's any different between this and a {{#author: }} parser function apart from the inability to access it from the wikitext? As noted, it's perfectly possible for the data to be in a separate field on the upload form, either by default or by per-wiki hackery. This is likely to result in as many "why can't I edit the bits of wikitext which diff, history, transclusion (let's not forget the enormous can of worms mucking around with the wikitext will open up there), etc assure me is there??" questions as it solves "what does this brace structure do?" ones. --HM

On 01/22/2011 08:15 PM, Bryan Tong Minh wrote: I disagree. In real life, there are always more compliated cases, where an author is not an author, but two authors or a sculptor, or one painter and one photographer. These things never fit in a single "author" field, and the same goes for any other separated fields. But the free-form Wikipedia can handle all real-world cases in plain human language. Various "expert systems" based on "artificial intelligence" existed since the 1980s, but none of them produced a universal encyclopedia. Only the text-based Wikipedia did. After this humiliating fact, the same AI people (now dressed as "semantic web" scholars) come and claim that they too could have built Wikipedia, if it only were more structured. They are wrong, of course. Lack of structure is precisely what built Wikipedia. -- Lars Aronsson (lars(a)aronsson.se) Aronsson Datateknik - http://aronsson.se

* Lars Aronsson &lt;lars(a)aronsson.se&gt; [Mon, 24 Jan 2011 07:06:02 +0100]: obscure One may have not just a single triple for that, but the list / set of triples for the same person in a different role (different kind of author). There are two extremes - not to have any structure or to be overly structural. If there are few extra fields for an image description, why don't generalize it for all kinds of measured data - geographical, historical, population statistics, financial and economical data and so on? Why only the images are allowed to have structural and measurable data? However, I don't think that Wikipedia should have AI, because it requires huge computing power, and the problem is that AI algorithms are not efficient enough. To have the data structured is not a bad thing. It probably should not even try to do SPARQL, but offer these things to external sits. Don't make complex queries, leave it for offline tools / bots or toolserver. Semantic bots are a good idea - they might mine the data finding the cross-sets. It should be even lighter than SMW. However, I might be wrong. Dmitriy

Before I respond to the recent new ideas, concepts and suggestions. I'd like to explain a few things about the backend (atleast the way it's currently planned to be) The mw_authors table contains unique authors by either a name or a userid. And optionally a custom attribution can be given (fallback to authorname, user real_name or user_name) Also optionally a url can be given (fallback to nothing or userpage). The mw_license table contains the different licenses a wiki allows to be used. Their canonical name (eg. "GFDL", "CC-BY-SA-3.0" etc.), url to legal code and usage count[1]. mw_file_props is a table that keeps previous versions of file_props as well. And is linked to mw_revision by fp_id in rev_fileprops_id (like mw_text is linked in rev_text_id). Both authors and licenses are uniquely identified by their id. This makes it easy to change stuff later on in an AuthorManager (eg. different url, username change etc.). The texts and complete titles of the licenses are stored in interface messages (for internationalization). MediaWiki:License-<uniq>-text could for example contain {{Cc-by-sa-3.0|attribution=$2}} on Wikimedia Commons. - If we store the links in the wikitext (like {{#fileauthor:}} and {{#filelicense:}}, the advantages are basically two things: 1) It has all features of editing and revisioning (better history, edit conflict, diff view, etc.) 2) No need for a revisionized mw_file_props, we can store the current values in mw_page_props Possible down side is that a diff like - {{#fileauthor:2}} {{filelicense:12}} + {{#fileauthor:10}} {{#fileauthor:12}} {{#filelicense: doesn't mean very much. I.m.h.o The solution is not to store the actual names in wikitext so that the diffs are better, but to either not store it in wikitext at all, or customize the behaviour everywhere: * edit form: extract parserfunction calls from wikitext before anything else, and put it in seperate form elements * diff view: get the names of those authors and licenses and somehow include it in the diff view This could be done a bit like AbuseFilter's diff between filter versions (ie. before "Line 1", would be "Author" and "License") * saving form: convert back to {{#parserfunction:}} calls and prepending it to wikitext * action=raw: ? * action=render: ? * api-parse: ? right now I think storing it in wikitext and customizing it everywhere like shown above is not worth the trouble and would likely bring it's own troubles. Keeping it seperate from wikitext is more work once but I think it pays off. But again, nothing is final yet. Everything is possible. -- Krinkle [1]: The usage count (mw_license.lic_count) is a bit like edit count (increased/decreased when saving files)

* Michael Dale &lt;mdale(a)wikimedia.org&gt; [Mon, 24 Jan 2011 13:18:00 -0600]: for and Michael is correct - API module to extract data from already existing nested templates and to replace the data (when needed) probably is the only thing that is required to make Wikipedia more structural and semantical. Then, the whole collecting and analyzing of triples can be off-loaded to externals bots and tools. Great idea, imho. Dmitriy