[Wikipedia-l] Feature request

[Neil Harris]

Larry Sanger wrote:

 >Sorry if this is already in the hopper or (!) has already been done, but
 >it seems long overdue:
 >
 >We need a way to compile, based on lists of links (I guess), "Recent
 >Changes" lists for all articles about a general topic.  I think this is
 >actually a very pressing need that should have been taken care of long
 >ago.  (For the record, I asked for it probably a year ago or more.)  The
 >idea is that we'd be able to maintain lists of all articles on a given
 >general subject, such as philosophy, and any changes made to any articles
 >on the subject would show up on the recent changes page for that subject.
 >
 >Why do we need this?  At least one reason is it might help attract experts
 >to the project.  Just speaking for myself, I'm sure I'd spend more time on
 >Wikipedia if there were a philosophy recent changes page.  More
 >importantly, the recent changes page has *always* been huge.  It's now
 >more cumbersome than ever and makes it hard for people to focus their
 >attention, which would be a nice option.  Lack of the feature also makes
 >it hard to *monitor* goings-on in a general subject area.
 >
 >With the relatively new mysql-driven software, this shouldn't be as
 >difficult as it might have been before.  One could compile personal lists
 >using the "watch this page" feature; what I suggest is that we have
 >publicly-editable and publicly-viewable lists of the same nature.
 >
 >(Minor point: in a list of recent changes pages, I think there should be
 >automatically listed the number of topics that are listed under a given
 >subject.  That'd give us an idea of how much more work there is to be done
 >in adding to the list.)
 >
 >I am not committed to any particular version of the feature, by the way.
 >I'd just like to see it done.  I don't want to have to wade through 5000
 >edits just to see all the recent philosophy edits.
 >
 >Larry
 >
 >
 >[Wikipedia-l]
 >To manage your subscription to this list, please go here:
 >http://www.nupedia.com/mailman/listinfo/wikipedia-l
 >
 >
 >
 >
 >
I have some ideas on this:

The problem of finding subject groups is closely related to the problem
of indexing.

The problem with the current link structure is that it is much too
dense: you can get between articles very easily, as Wikipedia is a very
"small world" network. This tends to defeat any attempts at automated
indexing. What is needed is a way of making some pages and links more
visible than others to automatic indexing systems.

We define a new "category" namespace. An article can contain any number
of "category" links, which do not appear in the main body of the
article, but instead in a separate area, like the inter-language links.
There, they link to a placeholder "category" page which can be used to
define and describe the category. (And its "category talk" page can be
used to discuss the category).

Now, all pages that link to a category page "belong" to that category.
Categories are just sets: there can be any number of them, and they can
belong to multiple competing schemes.  And categories can belong to
categories, too, allowing for hierarchies and networks of categories to
be created. The presence of categories will make machine indexing much
easier.

Some very basic categories that might be useful for a start:

*[[category:animal]] eg wolf, cat, bird, dinosaur
*[[category:vegetable]]  eg potato, cactus
*[[category:person]] eg Isaac Newton, Sherlock Holmes (but see below)
*[[category:time period]] eg 20th century, Feburary, 200 BC, Cenozoic
*[[category:event]]  eg Wars of the Roses, 1997 Academy Awards
*[[category:place]] eg Dubrovnik, Alaska, Indian Ocean, Atlantis (but
see below)
*[[category:field of study]] eg Biology, Chemistry, Philosophy, Law,
Accountancy, Civil engineering
** not sure of the best name(s) for this: field of endeavour, subject of
inquiry?
*[[category:fictional]] eg Sherlock Holmes, Atlantis
*[[category:abstraction]] eg Soul, Mind, Sophie Germain prime,
Mathematical set

I'd like to get these very simple categories in place first, as a sort
of "page coloring" experiment. Notice that they are neither complete,
nor framed in the form of a hierarchy: this is not a taxonomy. For
example, "prion" belongs to none of these categories. Perhaps someone
will create a [[category:other lifeform]] page for the Archaea, prions
and viruses.

How to bootstrap the process? My first idea is this:

* assign categories to about 1000 articles by hand
* train a naive Bayesian classifier to recognise each category
* adjust thresholds to make sure that classifications are reasonably
accurate
* machine-classify the entire Wikipedia!

Now, this process will be less than perfect. Some articles will be
mis-classified, others will be missed because the threshold
probabilities were set too high: ie both type I and type II errors.
Mis-classification will not damage any actual articles, it will only
result in errors in machine-generated indexes. But at this point, manual
editing will take over.

New articles can be machine-classified once they reach say 250
characters, using a Bayesian classifier that is trained on the corpus as
a whole: and again, once they have been machine-classified once, they
are then left alone thereafter.

Now, at this point, we may not need to create a "philosophy" or
"chemistry" category. Instead, we can just note that these are
[[category:field of study|fields of study]] and hence that pages that
link to them, or are linked from them, "belong" to them in some sense.
Similar treatment can be done for time periods.

I'm not 100% sure how this would work, but I think that a workable
mechanism could be evolved, given the initial category coloring.

Neil