Dear all
I don't think this is a difficult problem, two points should be clarified first:
1. Al most all facts on Wikipedia need not be sourced.
2. Since sourcing cannot inform us of truth/falsehood of a fact - it, at best, indicates the authority of its source.
I am not a lawyer, only a "Information Specialist." So this following should be double checked with legal:
Fact: Google, Caches practically anything it indexes:
However attempts by some site owners to claim this cache as a violation of their copyright have been consistently repealed/dismissed by US law courts.
I am not even sure what legal grounds we used, but caching is considered part of web tech and e.g. the browser caches are also not considered copyright violations.
On the drawing board of the Next Generation Search Engine is "content analytics" capability for doing authority assessment of references using:
1. A Transitive Bibliometric authority model.
2. A metric of reference longevity. (Fad vs. Fact test)
3. Bootstrapping using content analysis where access to full text of source was made available.
While the above model is complex, it would take (me) about 2 weeks of work to set up prototype reference repository --
a Nutch (crawler) + Solr + Storage (say MySql/Hbase/Cassandra) combination to index:
external links,
references including urls,
references with no urls.
This data would be immediately consumable via http using standards based request. (A SOLR feature).
To add Integration with existing Search UI would probably take another 2 weeks. As would adding support for caching/indexing most significant non html document formats.
However It would not be able to access content behind pay walls without access to a password. If and only if WMF sanctions this strategy - I could also draft a 'win win' policy for Encouraging such "Hidden Web" resource owners to provide free access to such a crawler. And possibly even open up their pay walls to our editors .
e.g. remove "No Follow" directive from links to high WP:RS partners.
I hope this help.
Oren Bochman.
MediaWiki Search Developer.
From: wikidata-l-bounces@lists.wikimedia.org [mailto:wikidata-l-bounces@lists.wikimedia.org] On Behalf Of John Erling Blad Sent: Sunday, April 01, 2012 10:01 PM To: Discussion list for the Wikidata project. Subject: Re: [Wikidata-l] Archiving references for facts?
Archiving a page should be pretty safe as long as the archived copy is only for internal use, that means something like OTRS. If the archived copy is republished it _might_ be viewed as a copyright infringement.
Still note that the archived copy can be used for automatic verification, ie extract a quote and check that against a stored value, without infringing any copyright. If a publication is withdrawn it might be an indication that something is seriously wrong with the page, and no matter what the archived copy at WebCitation says the page can't be trusted.
Its really a very difficult problem.
Jeblad
On 1. apr. 2012 14.08, "Helder" helder.wiki@gmail.com wrote: