Hi Brian,
I'm not sure this is foundation-l type of discussion, but let me give a couple of comments. I took the liberty of re-running your sample query "hippie" using google and built-in search on simple.wp, here are the results I got for top 10 hits:
Google: [1] Hippie, Human Be-In, Woodstock Festival, South Park, Summer of Love, Lysergic acid diethylamide, Across the Universe (movie), Glam rock, Wikipedia:Simple talk/Archive 27, Morris Gleitzman
simple.wikipedia.org: [2] Hippie, Flower Power, Counterculture, Human Be-In, Summer of Love, Woodstock Festival, San Francisco California, Glam Rock, Psychedelic pop, Neal Cassady
LDA (your method results from your e-mail): Acid rock, Aldeburgh Festival, Anne Murray, Carl Radle, Harry Nilsson, Jack Kerouac, Phil Spector, Plastic Ono Band, Rock and Roll, Salvador Allende, Smothers brothers, Stanley Kubrick
Personally, I think the results provided by the internal search engine are the best, maybe even slightly better than google's, and I'm not sure what kind of relatedness LDA captures.
If we were to systematically benchmark these methods on en.wp I think google would be better than internal search, mainly because it can extract information from pages that link to wikipedia (which apparently doesn't work as well for simple.wp). But that is beside the point here.
I think it is interesting that you found that certain classes of pages (e.g. featured articles) could be predicted from some statistical properties, although I'm not sure how big is your false discovery rate.
In any case, if you do want to work on improving the search engine and classification of articles, here are some ideas I think are worth pursuing and problems worth solving:
* integrating trends into search results - if one searches for "plane crash" a day after a plane crashes, he should get first hit that plane crash and not some random plane crash from 10 years ago - we can conclude this is the one he wants because it is likely that this page is going to get a lots of page hits. So, this boils down to: integrate page hit data into search results in a way that is robust and hard to manipulate (e.g. by running a bot or refreshing a page million times)
* morphological and context-dependent analysis, if a user enters a query like "douglas adams book" what are the concepts in this query? Should we group the query like [(douglas adams) (book)] or [(douglas) (adams book)]? Can we devise a general rule that will quickly and reliably separate the query into parts that are related to each other, and then use those to search through the article space to find the most relevant articles?
* technical challenges: can we efficiently index expanded article with templates, can we make efficient category intersection (w/o subcategories)
* extracting information: what kinds of information is in wikipedia, how do we properly extract it and index it? What about chemical formulas, geographical locations, computer code, stuff in templates, tables, image captions, mathematical formulas....
* how can we improve on the language model? Can we have smarter stemming and word disambiguation (compare shares in "shares and bonds" vs "John shares a cookie"). What about synonyms and acronyms? Can we improve on the language model "did you mean..." is using to correlate related words?
Hope this helps,
Cheers, robert (a.k.a "the search guy")
[1] http://www.google.co.uk/search?q=hippie+site%3Asimple.wikipedia.org [2] http://simple.wikipedia.org/w/index.php?title=Special%3ASearch&search=Hi...
Brian J Mingus wrote:
This paper (first reference) is the result of a class project I was part of almost two years ago for CSCI 5417 Information Retrieval Systems. It builds on a class project I did in CSCI 5832 Natural Language Processing and which I presented at Wikimania '07. The project was very late as we didn't send the final paper in until the day before new years. This technical report was never really announced that I recall so I thought it would be interesting to look briefly at the results. The goal of this paper was to break articles down into surface features and latent features and then use those to study the rating system being used, predict article quality and rank results in a search engine. We used the [[random forests]] classifier which allowed us to analyze the contribution of each feature to performance by looking directly at the weights that were assigned. While the surface analysis was performed on the whole english wikipedia, the latent analysis was performed on the simple english wikipedia (it is more expensive to compute). = Surface features = * Readability measures are the single best predictor of quality that I have found, as defined by the Wikipedia Editorial Team (WET). The [[Automated Readability Index]], [[Gunning Fog Index]] and [[Flesch-Kincaid Grade Level]] were the strongest predictors, followed by length of article html, number of paragraphs, [[Flesh Reading Ease]], [[Smog Grading]], number of internal links, [[Laesbarhedsindex Readability Formula]], number of words and number of references. Weakly predictive were number of to be's, number of sentences, [[Coleman-Liau Index]], number of templates, PageRank, number of external links, number of relative links. Not predictive (overall - see the end of section 2 for the per-rating score breakdown): Number of h2 or h3's, number of conjunctions, number of images*, average word length, number of h4's, number of prepositions, number of pronouns, number of interlanguage links, average syllables per word, number of nominalizations, article age (based on page id), proportion of questions, average sentence length. :* Number of images was actually by far the single strongest predictor of any class, but only for Featured articles. Because it was so good at picking out featured articles and somewhat good at picking out A and G articles the classifier was confused in so many cases that the overall contribution of this feature to classification performance is zero. :* Number of external links is strongly predictive of Featured articles. :* The B class is highly distinctive. It has a strong "signature," with high predictive value assigned to many features. The Featured class is also very distinctive. F, B and S (Stop/Stub) contain the most information. :* A is the least distinct class, not being very different from F or G. = Latent features = The algorithm used for latent analysis, which is an analysis of the occurence of words in every document with respect to the link structure of the encyclopedia ("concepts"), is [[Latent Dirichlet Allocation]]. This part of the analysis was done by CS PhD student Praful Mangalath. An example of what can be done with the result of this analysis is that you provide a word (a search query) such as "hippie". You can then look at the weight of every article for the word hippie. You can pick the article with the largest weight, and then look at its link network. You can pick out the articles that this article links to and/or which link to this article that are also weighted strongly for the word hippie, while also contributing maximally to this articles "hippieness". We tried this query in our system (LDA), Google (site:en.wikipedia.org hippie), and the Simple English Wikipedia's Lucene search engine. The breakdown of articles occuring in the top ten search results for this word for those engines is: * LDA only: [[Acid rock]], [[Aldeburgh Festival]], [[Anne Murray]], [[Carl Radle]], [[Harry Nilsson]], [[Jack Kerouac]], [[Phil Spector]], [[Plastic Ono Band]], [[Rock and Roll]], [[Salvador Allende]], [[Smothers brothers]], [[Stanley Kubrick]]. * Google only: [[Glam Rock]], [[South Park]]. * Simple only: [[African Americans]], [[Charles Manson]], [[Counterculture]], [[Drug use]], [[Flower Power]], [[Nuclear weapons]], [[Phish]], [[Sexual liberation]], [[Summer of Love]] * LDA & Google & Simple: [[Hippie]], [[Human Be-in]], [[Students for a democratic society]], [[Woodstock festival]] * LDA & Google: [[Psychedelic Pop]] * Google & Simple: [[Lysergic acid diethylamide]], [[Summer of Love]] ( See the paper for the articles produced for the keywords philosophy and economics ) = Discussion / Conclusion = * The results of the latent analysis are totally up to your perception. But what is interesting is that the LDA features predict the WET ratings of quality just as well as the surface level features. Both feature sets (surface and latent) both pull out all almost of the information that the rating system bears. * The rating system devised by the WET is not distinctive. You can best tell the difference between, grouped together, Featured, A and Good articles vs B articles. Featured, A and Good articles are also quite distinctive (Figure 1). Note that in this study we didn't look at Start's and Stubs, but in earlier paper we did. :* This is interesting when compared to this recent entry on the YouTube blog. "Five Stars Dominate Ratings" http://youtube-global.blogspot.com/2009/09/five-stars-dominate-ratings.html:... I think a sane, well researched (with actual subjects) rating system is well within the purview of the Usability Initiative. Helping people find and create good content is what Wikipedia is all about. Having a solid rating system allows you to reorganized the user interface, the Wikipedia namespace, and the main namespace around good content and bad content as needed. If you don't have a solid, information bearing rating system you don't know what good content really is (really bad content is easy to spot). :* My Wikimania talk was all about gathering data from people about articles and using that to train machines to automatically pick out good content. You ask people questions along dimensions that make sense to people, and give the machine access to other surface features (such as a statistical measure of readability, or length) and latent features (such as can be derived from document word occurence and encyclopedia link structure). I referenced page 262 of Zen and the Art of Motorcycle Maintenance to give an example of the kind of qualitative features I would ask people. It really depends on what features end up bearing information, to be tested in "the lab". Each word is an example dimension of quality: We have "*unity, vividness, authority, economy, sensitivity, clarity, emphasis, flow, suspense, brilliance, precision, proportion, depth and so on.*" You then use surface and latent features to predict these values for all articles. You can also say, when a person rates this article as high on the x scale, they also mean that it has has this much of these surface and these latent features.
= References =
- DeHoust, C., Mangalath, P., Mingus., B. (2008). *Improving search in
Wikipedia through quality and concept discovery*. Technical Report. PDFhttp://grey.colorado.edu/mediawiki/sites/mingus/images/6/68/DeHoustMangalathMingus08.pdf
- Rassbach, L., Mingus., B, Blackford, T. (2007). *Exploring the
feasibility of automatically rating online article quality*. Technical Report. PDFhttp://grey.colorado.edu/mediawiki/sites/mingus/images/d/d3/RassbachPincockMingus07.pdf _______________________________________________ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l