On Tue, Dec 2, 2008 at 2:57 PM, Aryeh Gregor <Simetrical+wikilist(a)gmail.com> wrote: Your're right. Fixed in the tool. It seems the JOIN query is significantly faster when all categories are large. However, with one or more small categories, I can do a pre-selection of pages (get page_ids for the intersection of the small categories, then look only for these in the larger ones), which in turn is significantly faster than the JOIN. My tool now uses the algorithm appropriate for the respective query. I am uncertain how much the toolserver factors in here. The poor thing is under a lot of stress ;-) I ran it again now, and it falls back to the JOIN solution, taking ~10 sec. As a worst-case scenario, I call that acceptable for the tool. It might not be acceptable for Wikipedia ATM. We could experiment how this performs on the "real" servers, though. Also, we could restrict certain queries. We know the category size, and in my approach, we know how many articles are in the "small category" intersection. Form there, we could guesstimate the worst-case time, and kill the query, or run it in MySQL slow mode (forgot the correct name) to not stress the servers too much. No matter what method, I think the problem should get high priority. I currently see a case on Commons, where there's now "Category:Paintings by Vincent van Gogh in this-and-that-museum". It's getting ridiculous (or is already there). Magnus P.S.: Just got a message on my Commons talk page about the "+incategory:" search function. * This is based on the Lucene index, right? How often is that updated? * Is there a decent interface/special page for that? It's a pain to enter this manually, and I doubt many people know about it * Is there a machine-readable interface for this? One that will return 5K hits without screenscraping?

On Tue, Dec 2, 2008 at 5:40 PM, Magnus Manske <magnusmanske(a)googlemail.com> wrote: api.php?list=search?

Then I suggest that Magnus immediately stops working on it, or else his curse of never getting anything into the core might strike ;-) -- [[en:User:Dschwen]] [[de:Benutzer:Dschwen]] [[commons:User:Dschwen]]

True, but is gems never seem to get enabled... -- [[en:User:Dschwen]] [[de:Benutzer:Dschwen]] [[commons:User:Dschwen]]

Aryeh Gregor <Simetrical+wikilist@...> writes: I'm with you - we've shown feasibility in large datasets with a lucene based approach, and I think we need to roll it out and test it with real users on real data. We need a new lucene index and a user interface (needs to be defined) suitable for average users to find useful. I'm thinking of a "browse related categories" type of function. Best Regards, Aerik

Daniel Schwen wrote: Users of the Swedish Wikipedia are increasingly starting to use Duesentrieb's CatScan tool. It is really useful, but could need some further improvement, especially in the handling of large categories. After a lengthy discussion (over many years) about category:tennis players and category:female tennis players in the Swedish Wikipedia, I created in late August 2008 the category:men and category:women, so that all profession categories could be freed from the burden of also documenting the gender. The Swedish Wikipedia still has a category:Danish tennis players (combining profession and nationality), just like the English Wikipedia, but gender is now documented separately, as in the German Wikipedia. All three languages have a category:1942 births. I think no language of Wikipedia has a combined category for tennis players born in 1942. So the question of atomic categories is not an absolute. It is more or less implemented everywhere. For finding tennis players born in 1942, even the English Wikipedia needs to do cross sectioning of categories. Radically changing the categorization system is not realistic. It was a huge effort already to introduce men/women in the Swedish Wikipedia, even though this was just adding categories (not removing any), and even though Swedish is not among the largest 10 Wikipedias. Within 3 months (September-November), some 75,000 articles were categorized, of which 15,000 women and 60,000 men. The ratio 1:4 (1 woman for every 4 men) is far more equal than the 1:6 ratio of the German Wikipedia. What I discovered then was that of these 75,000 biographies, only 60,000 were categorized according to year of birth. So we now have to birth categorize 15,000 articles before we can compile reliable statistics on how the gender imbalance shifts over time. Early estimates show that there is a 1:10 gender ratio in the 18th century and a 1:3 ratio for those born in the 1970s. So the larger imbalance (1:6) of the German Wikipedia might be explained by having a larger amount of 18th century biographies. -- Lars Aronsson (lars(a)aronsson.se) Aronsson Datateknik - http://aronsson.se

On Tuesday 02 December 2008 17:01:30 Daniel Schwen wrote: Perhaps just category intersection isn't enough. I was thinking about a tool that would allow intersection of various article data, including the categories. For example, suppose that I am maintaining [[1991 in art]]. I would want to find all articles that link to [[1991]] and are in a subcategory of [[:Category:Visual arts]]. And don't even get me started about what could be done if template parameters would be recorder somewhere...

We had a pretty lengthy discussion about this before the summer, and the consensus seemed to be that a fulltext-based approach looked most viable. I actually wrote an extension that does that, and promised to release it soon; that was quite a few months ago, and I never got around to it. I'll release it properly when I have time, which will hopefully be before Christmas :D The code needs some tweaking and refactoring, though. It's pretty tightly integrated with the article text search (both functions in one form) and has all kinds of weird features, because the guy who paid me to write it wanted them. It also doesn't support three-letter word searching (which core does these days, using a prefix hack), which is pretty bad since categories with short titles (or stopword titles) won't be found either. Roan Kattouw (Catrope)

Daniel Schwen schreef: Err.. what? Please explain what you mean by that. Without addressing Commons in particular, having an efficient way to get pages in the intersection of multiple categories would allow wikis to delete a category such as [[Category:Deceased Presidents of the United States]] and replace it by, say, [[Intersection:Deceased Presidents of the United States]], which would list all articles in [[Category:Deceased people]] and [[Category:Presidents of the United States]]. My extension alone doesn't make that possible, but it makes implementing such a feature considerably easier. Would you care to elaborate on what those unsolved problems are? It would be nice if you didn't judge people as naive rightaway. Roan Kattouw (Catrope)

2008/12/3 Daniel Schwen &lt;lists(a)schwen.de&gt;de>: If the machinery is in place to replace the present ridiculous sub-sub-sub-categories with something that *does their job just as well*, they'll die in quite reasonable order. If the machinery can't completely replace them without editor pain, it'll fail. If it can, it won't and Commons will be ENORMOUSLY happy 'cos we can then go wild treating cats like tags! - d.

I'd like for you to be right. But switching from the present category system to atomic categories is not as straight forward as having a few bots run over all existing cats. It will require an enormous amount of work. And so far I have not met willingness to change anything. Greg has shown a long time ago that fast category intersection is doable, but the echo has been pretty much zip, nada. Just note that simply replacing a category with all of it super categories is a dead end. You wouldn't believe the twists and turns in the category tree. Amusing example have been posted on this list already. So, yeah, sorry for my tone. I've pretty much kept my cool for the last N incarnations of this debate, but after repeating all the arguments for atomic cats and intersections and seeing zero improvement I'm getting a little frustrated. Call it "empiric evidence" rather than "assuming people to be naive" ;-) -- [[en:User:Dschwen]] [[de:Benutzer:Dschwen]] [[commons:User:Dschwen]]

Uhm, yeah.. except that intersection of atomic categories are not vaporware. We had proofs of concept for that and the interest was marginal. In any case. If someone would really just shoved it into mw core and enabled it on all the wmf sites I'd be happy. I concur that it would make the job convincing useres of a less retarded categorization scheme a bit easier. As far as Aeriks soapboxing from a few emails back goes: Let's not kid ourselves, tag based categorization is standard on commercial sites such as stockphotography libraries. We are not exactly inventing this... I'll shut up now, and I really hope that this is the last time we're having this discussion... (but boy, you will get an earfull if it isn't ;-) ) -- [[en:User:Dschwen]] [[de:Benutzer:Dschwen]] [[commons:User:Dschwen]]

On Wed, Dec 3, 2008 at 8:12 PM, David Gerard &lt;dgerard(a)gmail.com&gt; wrote: With a JS hack I had my tool integrated to the site. The AJAX calls went to the toolserver, but as far as the users could see it was running on the site. No one cared: It didn't produce useful results because of how categories are used, and when I suggested changing people just waved their arms at me "just make it walk the tree".

Gregory Maxwell wrote: That _is_ curious. When did this happen? It seems I also blinked and missed it. -- Ilmari Karonen

On Wed, Dec 3, 2008 at 7:12 PM, Daniel Schwen &lt;lists(a)schwen.de&gt; wrote: Vaporware with proofs of concept is still vaporware. The definition of vaporware is more or less something that doesn't go *beyond* proofs of concept. Category intersection has never been added to the software and there's no timetable for adding it to the software, so doing any recategorization right *now* to aid category intersection would be pointless. JS thingies may have been enabled on some wikis for some time periods, but that's very different from a feature being prominently added to *all* wikis. On Wed, Dec 3, 2008 at 8:16 PM, Gregory Maxwell &lt;gmaxwell(a)gmail.com&gt; wrote: What was the interface like (how noticeable/obtrusive), how long was it up, and why did it get removed? You're certainly going to need a critical mass of people who know about it and use it before there will be any effect. And enabling it on all wikis at once would likely help, too: if Germans get used to using it on dewiki and find it useful, they'll be more likely to push for it to be made useful on Commons. On Thu, Dec 4, 2008 at 7:45 AM, Tim Landscheidt &lt;tim(a)tim-landscheidt.de&gt; wrote: A reasonable point. In the medium term it could be handled by (you guessed it) bots. In the longer term, allowing people to define more concrete semantic relationships between categories (e.g., "X is partitioned into X1, X2, ..., Xn") could make this automatic within the software itself. In the end, all of these objections are really irrelevant to the technical issues here. The fact of the matter is that category intersection is widely supported in other major software products (in the form of tag intersection), it's something that a lot of people want, and so it would be good if it were in the core software. How fully various specific communities would want to use it is up to them -- that some communities might never choose to use a particular feature doesn't mean that it shouldn't be developed (cf. FlaggedRevs, etc.).

"Aryeh Gregor" &lt;Simetrical+wikilist(a)gmail.com&gt; wrote: Add to that the maintenance costs because you would want to ensure that if someone who is not aware of the concept of atomic categories adds a [[Category:Manhattan]] to something he adds [[Category:New York]], [[Category:East Coast of the United States]], [[Category:United States]] and the other gigazillion umbrella categories as well so searches for a building in a country bordering a water body will still show results. Tim

David Gerard wrote: If we're going to end up with hundreds of categories on each page, why not make the software automatically add all parent categories? It would fill the categorylinks table*, but it would as well by manually adding them. It would also require forcing the categories to be a graph and maybe limiting the number of parent categories, as to reduce a bit how expensive category position changes can be. But, if we leave that to 'manual actions', the same actions would be performed by bots, leading to the same cost and partially less coherent structure. *Add a expandedcategorylinks table? Probably also add a 'don't inherit' flag on categary table which can be appplied to high level categories such as 'All licenses' or 'Commons root'.

Gregory Maxwell wrote: True, but there are _some_ relationships that should always hold. All dogs are animals. All integers are numbers. All places in New York are in the United States. Arguably, any page which is in [[Category:Dogs]] but not in [[Category:Animals]] is a failure of atomic categorization. Of course, there are also many relationships that _don't_ hold so strictly. Most dogs are pets, but not all. Most places in the United States are in North America, but not all. So, yes, some of the consistency checking will have to be done at least partly manually. But really, I wouldn't worry about this too much. Sure, having a way to enforce some category relationships would be useful, as would automatically recommending others. But even if we don't implement it immediately in the software, someone will write a bot (or several) to help with it. It won't be perfect, but I wouldn't expect it to be much more broken than the current interlanguage link system, which we consider useful enough to keep deployed despite its numerous failings. (While thinking about this, I thought back to an earlier discussion on this list (or possibly wikien-l, can't remember now) about the fact that there are essentially two types of categories: thematic and taxonomic. For the former, the "tag" model of atomic categorization is quite natural, but the latter would fit much more naturally into a strictly hierarchical model. It might not be an entirely unreasonable idea to formally split the two, perhaps even into separate namespaces, and apply different technical approaches to handling them.) -- Ilmari Karonen