In a message dated 12/10/2010 2:58:08 PM Pacific Standard Time,
my idea was that you will want to search pages that
are referenced by
wikipedia already, in my work on kosovo, it would be very helpful
because there are lots of bad results on google, and it would be nice
to use that also to see how many times certain names occur.
That is why we need also our own indexing engine, I would like to
count the occurances of each term and what page they occur on, and to
xref that to names on wikipedia against them. Wanted pages could also
be assisted like this, what are the most wanted pages that match
against the most common terms in the new refindex or also existing
Well then all you would need to do is cross-reference the refs themselves.
You don't need to cache the underlying pages to which they refer.
So in your new search engine, when you search for "Mary, Queen of Scots"
you really are saying, show me those external references, which are mentioned,
in connection with Mary Queen of Scots, by Wikipedia.
That doesn't require caching the pages to which refs refer. It only
requires indexing those refs which currently are used in-world.