[Foundation-l] Improving search in Wikipedia through quality and concept discovery

Sun Nov 1 16:15:17 UTC 2009

Hi Brian,

I'm not sure this is foundation-l type of discussion, but let me give a 
couple of comments.
I took the liberty of re-running your sample query "hippie" using google 
and built-in search on simple.wp, here are the results I got for top 10 
hits:

Google:  [1]
Hippie, Human Be-In, Woodstock Festival, South Park, Summer of Love, 
Lysergic acid diethylamide, Across the Universe (movie), Glam rock, 
Wikipedia:Simple talk/Archive 27, Morris Gleitzman

simple.wikipedia.org: [2]
Hippie, Flower Power, Counterculture, Human Be-In, Summer of Love, 
Woodstock Festival, San Francisco California, Glam Rock, Psychedelic 
pop, Neal Cassady

LDA (your method results from your e-mail):
Acid rock, Aldeburgh Festival, Anne Murray, Carl Radle, Harry Nilsson, 
Jack Kerouac, Phil Spector, Plastic Ono Band, Rock and Roll, Salvador 
Allende, Smothers brothers, Stanley Kubrick

Personally, I think the results provided by the internal search engine 
are the best, maybe even slightly better than google's, and I'm not sure 
what kind of relatedness LDA captures.

If we were to systematically benchmark these methods on en.wp I think 
google would be better than internal search, mainly because it can 
extract information from pages that link to wikipedia (which apparently 
doesn't work as well for simple.wp). But that is beside the point here.

I think it is interesting that you found that certain classes of pages 
(e.g. featured articles) could be predicted from some statistical 
properties, although I'm not sure how big is your false discovery rate.

In any case, if you do want to work on improving the search engine and 
classification of articles, here are some ideas I think are worth 
pursuing and problems worth solving:

* integrating trends into search results - if one searches for "plane 
crash" a day after a plane crashes, he should get first hit that plane 
crash and not some random plane crash from 10 years ago - we can 
conclude this is the one he wants because it is likely that this page is 
going to get a lots of page hits. So, this boils down to: integrate page 
hit data into search results in a way that is robust and hard to 
manipulate (e.g. by running a bot or refreshing a page million times)

* morphological and context-dependent analysis, if a user enters a query 
like "douglas adams book" what are the concepts in this query? Should we 
group the query like [(douglas adams) (book)] or [(douglas) (adams 
book)]? Can we devise a general rule that will quickly and reliably 
separate the query into parts that are related to each other, and then 
use those to search through the article space to find the most relevant 
articles?

* technical challenges: can we efficiently index expanded article with 
templates, can we make efficient category intersection (w/o subcategories)

* extracting information: what kinds of information is in wikipedia, how 
do we properly extract it and index it? What about chemical formulas, 
geographical locations, computer code, stuff in templates, tables, image 
captions, mathematical formulas....

* how can we improve on the language model? Can we have smarter stemming 
and word disambiguation (compare shares in "shares and bonds" vs  "John 
shares a cookie"). What about synonyms and acronyms? Can we improve on 
the language model "did you mean..." is using to correlate related words?

Hope this helps,

Cheers, robert (a.k.a "the search guy")

[1] http://www.google.co.uk/search?q=hippie+site%3Asimple.wikipedia.org
[2] 
http://simple.wikipedia.org/w/index.php?title=Special%3ASearch&search=Hippie&fulltext=Search

Brian J Mingus wrote:
> This paper (first reference) is the result of a class project I was part of
> almost two years ago for CSCI 5417 Information Retrieval Systems. It builds
> on a class project I did in CSCI 5832 Natural Language Processing and which
> I presented at Wikimania '07. The project was very late as we didn't send
> the final paper in until the day before new years. This technical report was
> never really announced that I recall so I thought it would be interesting to
> look briefly at the results. The goal of this paper was to break articles
> down into surface features and latent features and then use those to study
> the rating system being used, predict article quality and rank results in a
> search engine. We used the [[random forests]] classifier which allowed us to
> analyze the contribution of each feature to performance by looking directly
> at the weights that were assigned. While the surface analysis was performed
> on the whole english wikipedia, the latent analysis was performed on the
> simple english wikipedia (it is more expensive to compute). = Surface
> features = * Readability measures are the single best predictor of quality
> that I have found, as defined by the Wikipedia Editorial Team (WET). The
> [[Automated Readability Index]], [[Gunning Fog Index]] and [[Flesch-Kincaid
> Grade Level]] were the strongest predictors, followed by length of article
> html, number of paragraphs, [[Flesh Reading Ease]], [[Smog Grading]], number
> of internal links, [[Laesbarhedsindex Readability Formula]], number of words
> and number of references. Weakly predictive were number of to be's, number
> of sentences, [[Coleman-Liau Index]], number of templates, PageRank, number
> of external links, number of relative links. Not predictive (overall - see
> the end of section 2 for the per-rating score breakdown): Number of h2 or
> h3's, number of conjunctions, number of images*, average word length, number
> of h4's, number of prepositions, number of pronouns, number of interlanguage
> links, average syllables per word, number of nominalizations, article age
> (based on page id), proportion of questions, average sentence length. :*
> Number of images was actually by far the single strongest predictor of any
> class, but only for Featured articles. Because it was so good at picking out
> featured articles and somewhat good at picking out A and G articles the
> classifier was confused in so many cases that the overall contribution of
> this feature to classification performance is zero. :* Number of external
> links is strongly predictive of Featured articles. :* The B class is highly
> distinctive. It has a strong "signature," with high predictive value
> assigned to many features. The Featured class is also very distinctive. F, B
> and S (Stop/Stub) contain the most information.
>  :* A is the least distinct class, not being very different from F or G. =
> Latent features = The algorithm used for latent analysis, which is an
> analysis of the occurence of words in every document with respect to the
> link structure of the encyclopedia ("concepts"), is [[Latent Dirichlet
> Allocation]]. This part of the analysis was done by CS PhD student Praful
> Mangalath. An example of what can be done with the result of this analysis
> is that you provide a word (a search query) such as "hippie". You can then
> look at the weight of every article for the word hippie. You can pick the
> article with the largest weight, and then look at its link network. You can
> pick out the articles that this article links to and/or which link to this
> article that are also weighted strongly for the word hippie, while also
> contributing maximally to this articles "hippieness". We tried this query in
> our system (LDA), Google (site:en.wikipedia.org hippie), and the Simple
> English Wikipedia's Lucene search engine. The breakdown of articles occuring
> in the top ten search results for this word for those engines is: * LDA
> only: [[Acid rock]], [[Aldeburgh Festival]], [[Anne Murray]], [[Carl
> Radle]], [[Harry Nilsson]], [[Jack Kerouac]], [[Phil Spector]], [[Plastic
> Ono Band]], [[Rock and Roll]], [[Salvador Allende]], [[Smothers brothers]],
> [[Stanley Kubrick]]. * Google only: [[Glam Rock]], [[South Park]]. * Simple
> only: [[African Americans]], [[Charles Manson]], [[Counterculture]], [[Drug
> use]], [[Flower Power]], [[Nuclear weapons]], [[Phish]], [[Sexual
> liberation]], [[Summer of Love]] * LDA & Google & Simple: [[Hippie]],
> [[Human Be-in]], [[Students for a democratic society]], [[Woodstock
> festival]] * LDA & Google: [[Psychedelic Pop]] * Google & Simple: [[Lysergic
> acid diethylamide]], [[Summer of Love]] ( See the paper for the articles
> produced for the keywords philosophy and economics ) = Discussion /
> Conclusion = * The results of the latent analysis are totally up to your
> perception. But what is interesting is that the LDA features predict the WET
> ratings of quality just as well as the surface level features. Both feature
> sets (surface and latent) both pull out all almost of the information that
> the rating system bears. * The rating system devised by the WET is not
> distinctive. You can best tell the difference between, grouped together,
> Featured, A and Good articles vs B articles. Featured, A and Good articles
> are also quite distinctive (Figure 1). Note that in this study we didn't
> look at Start's and Stubs, but in earlier paper we did. :* This is
> interesting when compared to this recent entry on the YouTube blog. "Five
> Stars Dominate Ratings"
> http://youtube-global.blogspot.com/2009/09/five-stars-dominate-ratings.html:*
> I think a sane, well researched (with actual subjects) rating system
> is
> well within the purview of the Usability Initiative. Helping people find and
> create good content is what Wikipedia is all about. Having a solid rating
> system allows you to reorganized the user interface, the Wikipedia
> namespace, and the main namespace around good content and bad content as
> needed. If you don't have a solid, information bearing rating system you
> don't know what good content really is (really bad content is easy to spot).
> :* My Wikimania talk was all about gathering data from people about articles
> and using that to train machines to automatically pick out good content. You
> ask people questions along dimensions that make sense to people, and give
> the machine access to other surface features (such as a statistical measure
> of readability, or length) and latent features (such as can be derived from
> document word occurence and encyclopedia link structure). I referenced page
> 262 of Zen and the Art of Motorcycle Maintenance to give an example of the
> kind of qualitative features I would ask people. It really depends on what
> features end up bearing information, to be tested in "the lab". Each word is
> an example dimension of quality: We have "*unity, vividness, authority,
> economy, sensitivity, clarity, emphasis, flow, suspense, brilliance,
> precision, proportion, depth and so on.*" You then use surface and latent
> features to predict these values for all articles. You can also say, when a
> person rates this article as high on the x scale, they also mean that it has
> has this much of these surface and these latent features.
>
> = References =
>
>
>    - DeHoust, C., Mangalath, P., Mingus., B. (2008). *Improving search in
>    Wikipedia through quality and concept discovery*. Technical Report.
> PDF<http://grey.colorado.edu/mediawiki/sites/mingus/images/6/68/DeHoustMangalathMingus08.pdf>
>    - Rassbach, L., Mingus., B, Blackford, T. (2007). *Exploring the
>    feasibility of automatically rating online article quality*. Technical
>    Report. PDF<http://grey.colorado.edu/mediawiki/sites/mingus/images/d/d3/RassbachPincockMingus07.pdf>
> _______________________________________________
> foundation-l mailing list
> foundation-l at lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
>
>   