On Sat, Feb 23, 2008, Tim Starling tstarling@wikimedia.org wrote:
But it seems to me, if you look at data storage software already in use, Lucene is much better suited for computing intersections than MySQL.
Tim, aren't you kind of the point guy for the lucene search? Would you be up for setting up a categories index? I don't know how the update works (I think, from what I've read, that it does a big index regeneration on some kind of schedule, but I really don't know).
I think it could be implemented as either a separate index, or as a new field on the current index.
I'd be happy to help, but I'm totally unfamiliar with the code, and don't really want to set up Java on my server for testing... I've created lucene indexes on the categories table before, but not in any way that even approaches a production type environment. Maybe that still leaves some opportunity to help though.
Best Regards, Aerik