On Thu, Feb 4, 2010 at 10:02 AM, Gregory Maxwell <gmaxwell(a)gmail.com> wrote: Right. Supporting category intersection and search in category with better UI (we already sort of support it if you know the right magic terms) is what we should be aiming for here.

On Thu, Feb 4, 2010 at 11:03 AM, Robert Stojnic <rainmansr(a)gmail.com> wrote: Yup. Any volunteers? My understanding is that right now, the backend supports category searches as long as the categories are spelled out literally in the wikitext (not via template). That's not a big restriction, so what we could really use right now is UI, which shouldn't require such specialized skills. So, does anyone want to: 1) Mock up basic UI for category intersections/search in category? 2) Implement it? After that we can talk about fancy things like automatically suggesting categories to intersect with or whatever . . . we don't even have the most basic UI right now.

On Thu, Feb 4, 2010 at 11:28 AM, Conrad Irwin <conrad.irwin(a)googlemail.com> wrote: No, probably not, but it would be even easier to not worry about it yet (unless someone wants to!). On Thu, Feb 4, 2010 at 11:37 AM, Daniel Schwen <lists(a)schwen.de> wrote: Correct. Automatic flattening is not good enough -- manual flattening is necessary. Maybe if we had a better category intersect feature, more wikis would do manual flattening. If they don't, I guess they won't get the feature. Automatic flattening is not a substitute.

This is putting the cart in front of the ox yet again. A few mails up Aryeh and Gregory both come to the conclusion that automatic flattening is useless. Yet category flattening would be a prerequisite to intersections. The only way to get proper intersection is manual flattening i.e. atomic categorization. As long as nobody is pushing commons _hard_ to change their categorization system _nothing_ will happen and we'll meet on this list again in about one year repeating the same discussion.

Ahh, the good old chicken and egg ;-) I don't let that count. We have plenty of working category intersection tools already. Their usefulness is limited however because the category system is so screwed up. The ball is definitely in the categorization-court!

Robert Stojnic schrieb: Wikimedia Germany has contracted Neil Harris to work on implementing deep category intersection. The goal is basically a rewrite of my sucky CatScan tool. The result is hopefully fast & generic enough so it can be used as a service that integrates with the current search infrastructure. The project has started, there is funding and a project plan. I expect to see usable results soon. In fact, I hope to present this at the developer meeting in april (neil, contact me about attending) and discuss the integration into lucene search. I agree that full recursive flattening of the current category structure leads to bad results some times (especially on the english wikipedia, commons is quite bad too), a depth of 5 however is generally useful. One common use case is intersecting a content category with a maintenance category, for organizing editorial work in a wiki project. In that case, at least one category comes from a template. Atomic categorization aka tagging however also sucks: the tags are either too generic (so it's hard to find stuff) or too specific (you never know what to search for). tags implying/including other tags is very useful. which is exactly what categories with deep intersection will provide. -- daniel

Magnus Manske schrieb: Yes, lots more options than my old thingy, thanks magnus :) but still bound to recursive calls to the database, which is what i really want to get rid of. the lookup needs to be snappy. -- daniel

On Thu, Feb 4, 2010 at 6:40 PM, Tim Landscheidt <tim(a)tim-landscheidt.de> wrote: Incremental updates for a 'flattened copy' aren't especially realistic... as one user operation can produce millions of operations on the server. I won't bother saying much more, Daniel Schwen pretty much speaks for my view.

On Fri, Feb 5, 2010 at 3:57 AM, Daniel Kinzler <daniel(a)brightbyte.de> wrote: And as for [[Category:People executed for heresy]] -> [[Category:Joan of Arc]] -> [[English claims to the French throne]]? That's only two steps, and it already doesn't make sense. You could argue that [[Category:Joan of Arc]] really means [[Category:Stuff related to Joan of Arc]] and shouldn't be in [[Category:People executed for heresy]], but that sounds like it would take as much recategorization work as just using atomic categories -- and much subtler.

On 6/02/10 6:44 AM, Tei wrote: Not at all, it's entirely reasonable to discuss the problems associated with the current categorisation system, and what methods we'd like to use to improve it. -- Andrew Garrett agarrett(a)wikimedia.org http://werdn.us

On Sun, Feb 7, 2010 at 7:01 AM, David Gerard <dgerard(a)gmail.com> wrote: Right now, I'd try just waiting. As Daniel pointed out in this thread, Neil Harris is already being paid to work on it.

Way to be acerbic... [..] Not really. There are two main points 1) category deep tree traversal (flattening / deep indexing) at runtime is technically unfeasable. 2) automatic flattening produces nonsense results Ok, lets's say Neil found a way to deal with 10. I give you that this is implementation specific. Number 2) however is independent of any implementation. Here you have your "hoop" (to to stick with your pejorative lingo): Get rid of the crazy category system and go atomic. What is vague about this, what part of this is unclear to you?

2010/2/7 David Gerard <dgerard(a)gmail.com>om>: There's no reason why it couldn't be the other way around: an intersection feature could be written and deployed *first*, *then* the category trees on Commons would be gradually migrated to the new system. Issues like nonsense results for automatic flattening could be migitated by disabling features or making them less visible. I did on my earlier reply, and I got a bounce from commons-l-owner saying my message was rejected because I'm not subscribed to commons-l. I'm not going to subscribe to that list, so I left the Cc: commons-l out this time. Roan Kattouw (Catrope)

On Sun, Feb 7, 2010 at 8:42 AM, David Gerard &lt;dgerard(a)gmail.com&gt; wrote: Nobody "demanded" this except possibly Daniel Schwen, who has never even committed anything to SVN outside of WikiMiniAtlas, let alone representing the opinion of The Developers. If you are incapable of distinguishing the personal opinion of a random person on this list from "demands" by "the developers", maybe you should unsubscribe and save everyone the trouble. That way you'll avoid annoying the developers by posting uninformed and obnoxious things like this, and avoid confusing non-developers by forwarding them irrelevant or incomprehensible wikitech-l posts out of context (and this is not the first time you've done that). On Sun, Feb 7, 2010 at 9:14 AM, Daniel Schwen &lt;lists(a)schwen.de&gt; wrote: Since when do we write features only for Commons? Some wikis already have atomic categories -- e.g., <http://de.wikipedia.org/wiki/Kategorie:Frau>. It would be a useful feature to any number of users regardless of what Commons does or does not do. In fact, it would be useful to Commons too even without atomic categorization, just not as useful as it could be. On the other hand, it's thoroughly unreasonable to expect any wiki to change how they do things based on technologies that have been talked about for years and may or may not materialize in the foreseeable future. No, stuff on the toolserver that isn't integrated into the interface (and doesn't have a very nice interface itself) doesn't count.

On Sun, Feb 7, 2010 at 11:50 AM, David Gerard &lt;dgerard(a)gmail.com&gt; wrote: *Nothing* said here is a "message being sent" to anyone other than developers and sysadmins. This is not a medium for official dev announcements to the larger world. If something actually needs to be announced to the projects, it will be announced to the projects, such as by global site notices or such. Taking it upon yourself to broadcast what's said here to the larger world is not useful, because what's said here is not targeted to the larger world. If you were actually a developer or followed MediaWiki development closely, you would understand perfectly well that Daniel Schwen's opinion doesn't count for anything in this particular case. He is not in a position to say "we won't do X", because he can't stop anyone else from doing it (and nor can most developers). CCing it to third parties as though it were an authoritative statement by The Developers is confusing and irresponsible. No demand *was* being made of the project, not by anyone who had any say. (No offense to Daniel -- if I said the same thing, my opinion wouldn't have any weight either.) If you don't understand that, then you don't know what you're reading here, and you should stop acting like you do. The same steps as for every other possible feature that anyone would want. Someone needs to implement the feature, commit it or get it committed, and address any objections that arise to avoid having it reverted or disabled. This includes jumping through any hoops that come up along the way -- and *nobody* knows what those are in advance. It looks like Neil Harris will be doing this over the next few months, with any luck, for this particular feature.

I think this thread is more heated than it needs to be. Perhaps we could keep the on-list discussion on-topic, and perhaps appropriate disclaimers and more judicious phrasing could be used to prevent people from getting the wrong idea. I also think that clarifying misunderstandings can be done without calling people ignorant (or implying it in the way a response is phrased). -- Andrew Garrett agarrett(a)wikimedia.org http://werdn.us

Daniel Kinzler wrote: When something starts to look overly complicated, perhaps we are addressing the wrong problem? If people really want flat tagging of photos in Wikimedia Commons, what stops anybody from inventing a simple template: {{tags|Sweden|sunset|swans}} ? The template could display this as a little box with some Javascript edit functionality (like HotCats). What happens when you click one of the tags? Maybe jump off to some toolserver application that has indexed all the tags in the previous database dump? Or go to a Google search? Googling for "swans sunset site:..." should bring up pictures that were tagged like that. Does it need to be more complicated than that? If so, exactly what is missing in my overly simplistic approach? -- Lars Aronsson (lars(a)aronsson.se) Aronsson Datateknik - http://aronsson.se