On 24 August 2010 14:57, emijrp <emijrp(a)gmail.com> wrote: +1

Here's the Archive's on-demand service: http://archive-it.org That would be the most reliable way to set up the partnership emijrp proposes.  And it's certainly a good idea.  Figuring out how to make it work for almost all editors and make it spam-proof may be interesting. SJ On Tue, Aug 24, 2010 at 8:45 PM, Ray Saintonge <saintonge(a)telus.net> wrote: -- Samuel Klein          identi.ca:sj           w:user:sj _______________________________________________ foundation-l mailing list foundation-l(a)lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

I've asked Gordon Mohr @ IA about how to work with archive-it. I will cc: this thread on any response. SJ On Tue, Aug 24, 2010 at 8:56 PM, George Herbert <george.herbert(a)gmail.com> wrote: wrote: times, dead or for the interesting external their needed. -- Samuel Klein identi.ca:sj w:user:sj _______________________________________________ foundation-l mailing list foundation-l(a)lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

Gordon @ IA was most friendly and helpful. archive-it is a subscription service for focused collections of sites; he had a different idea better suited to our work. Gordon writes: things its IA That sounds excellent to me, if possible (and I think close to what emijrp had in mind!) What would it take to produce such a feed? SJ PS - An aside: IA's policies include taking down any links on request, so this would not be a foolproof archive, but a 99% one. On Tue, Aug 24, 2010 at 9:13 PM, Samuel Klein <meta.sj(a)gmail.com> wrote: wrote: wrote: times, dead or for the interesting external improve their needed. https://lists.wikimedia.org/mailman/listinfo/foundation-l -- Samuel Klein identi.ca:sj w:user:sj _______________________________________________ foundation-l mailing list foundation-l(a)lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

*What would it take to produce such a feed?** * A real-time feed may or may not be the best idea, for several reasons. - One issue is that every edit would have to be examined not only for external links, but for external links that were not present previously. Doing this real-time may cause slowdowns or additional load for the servers - keep in mind that we would have to scan external links on all edits for all Wikipedia's; Counted together this would result in a very, very busy feed towards IA. - Sometimes added links are spam or otherwise not acceptable, which means they may be removed soon after. In such a case man would prefer not having them archived, since it would be a waste of time and work for IA. An alternate solution could be forwarding a list of new links every day. The Database Layout< http://upload.wikimedia.org/wikipedia/commons/4/41/Mediawiki-database-schem… Wikimedia seems to sugest that all external links are stored in a separate table in the database (And i presume this includes links in reference tags). I wonder if it would be possible to dump this entire table for IA, and afterwards send incremental change packages<http://en.wikipedia.org/wiki/Changeset>to them (Once a day perhaps?). That way they would always have a list of external links used by Wikipedia, and it would decrease the problem with performance hits, spam and links no longer used. If we only forwarded a feed with NEW links IA might end up with a long list of links which are removed over time. And above everything - the External Links table is simply a database table, which should be incredibly easy to read and process for IA, without custom coding required to read and store a feed. But perhaps the people at the tech mailing list have another \ better idea on how this should work :) ~Excirial On Sat, Aug 28, 2010 at 9:48 AM, Samuel Klein &lt;meta.sj(a)gmail.com&gt; wrote: would that be that is when https://lists.wikimedia.org/mailman/listinfo/foundation-l https://lists.wikimedia.org/mailman/listinfo/foundation-l _______________________________________________ foundation-l mailing list foundation-l(a)lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

A real-time feed of external links is overkill. As mentioned by others, the chief problem is linkrot of old links. All we need to do is dump the contents of externallinks.el_to from the database once a year or so, run a hex to ASCII conversion on it, zip it, and email it to someone at the Internet Archive. Anyone with access to the databases should be able to do this fairly easily. Rather than trying to engineer a complicated system that will take a year to implement, why not take this simple approach that will take care of 90+% of the problem? Ryan Kaldari

Nikola Smolenski wrote: this: > > - We are often copying free images or text from various sites (for > example flickr but other ones too). It happens that these sites go > offline or change their licenses later. Having such an archive, archived > by an independent organization, would be indisputable proof of copyright > status.

The internet archive [...] respects copyright and robots.txt and applies the directives retroactively.

No university, publisher, or newspaper has used my CC licensed images either commercially or non-commercially without checking with me first that the work is actually CC licensed. They have always carried out some form of due diligence to ascertain that the image is either licensed properly, and that they get a specific license to reuse. IOW they obtain a 'paper trail' of permission.

This looks like a solid organization. Solid in the sense that it wont go suddenly offline. Such links may be valuable for: - article references to sources, in case the source goes offline - article references to sources, in case thesource changes its content - media copies when the source changes or removes a license I took a look at the example in the french wiki, and didnt spot a date in the archive reference. If the source changes its content, this may pose a problem. kind regards, teun On Tue, Aug 24, 2010 at 3:57 PM, emijrp &lt;emijrp(a)gmail.com&gt; wrote: _______________________________________________ foundation-l mailing list foundation-l(a)lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

Hi, On Tue, Aug 24, 2010 at 4:47 PM, teun spaans &lt;teun.spaans(a)gmail.com&gt; wrote: of the archiving - I think the reason it is not displayed by default on the French Wikipedia is because the archive links are generated by JavaScript on the fly. (At least that was the case the last time I looked at the French Wikipedia.) Having the ability to store multiple copies of the same webpage (for different dates) was one of the first feature requests we had at the Hungarian Wikipedia and it seems they are working on it. Still, Wikiwix's service is very convenient and hassle free for all the static websites or references. Webcitation.org also has a service for on-demand archiving and they do store multiple versions of the same page. Unfortunately their service is often intermittent and their website tends to go dark, but otherwise it is a convenient service for manual archiving. (I had a bot once that sent each link through its service on the Hungarian Wikipedia, and for a time the English Wikipedia had a similar bot. Best regards, Bence _______________________________________________ foundation-l mailing list foundation-l(a)lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

On 24 August 2010 17:32, emijrp &lt;emijrp(a)gmail.com&gt; wrote: so in That's a key point: have multiple archives easily supported. (Hopefully not as complicated as what happens when you click on an ISBN.) - d. _______________________________________________ foundation-l mailing list foundation-l(a)lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

Google's cache only last a handful of days. It's very useful if the Website is down temporarily (for maintenance or overload for example) but totally useless otherwise.

2010/8/24 theo10011 &lt;de10011(a)gmail.com&gt; > Its a great idea, using the wayback machine to ward of link rot. I support > it but doesn't Google cache offer a similar service. there is also > deadURL.com which uses Google Cache, the Internet Archive, and user > submissions for gathering dead links. > > I would guess that Google Cache would have the highest and the longest > reliability, at least as long as Google exists, its their business. >

Does anyone know what the status of the Internet Archive is with respect to being a practical ongoing concern? In the last couple years IA has added relatively little web-based content. For example, their Wayback Machine currently offers: www.nytimes.com: 11 pages since 2006 en.wikipedia.org: 5 pages since 2008 www.nasa.gov: 12 pages since 2008 scienceblogs.com: 0 pages since 2008 It gives the impression that they are so ineffective at archiving recent content as to be effectively irrelevant. They do have a warning that it can take 6 or more months for newly accessed content to be incorporated into their database, but at this point the delay has been significantly more than that. Even at their peak they rarely archived more than a few hundred pages per major domain per year, which still amounts to a tiny fraction of the internet. The idea of seeking collaborations with people that archive web content is a good one, but I don't know that IA is really in a position to be all that useful. -Robert Rohde On Tue, Aug 24, 2010 at 6:57 AM, emijrp &lt;emijrp(a)gmail.com&gt; wrote: or the their needed. _______________________________________________ foundation-l mailing list foundation-l(a)lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l