This has reminded me to complain about Google Books. Google has the world's best OCR (in virtue of having the largest OCR'able dataset) and also has a mission to scan in all the public domain books they can get their hand on. They recently updated their interface to, as they put it, "make it easier to find our plain text versions of public domain books. If a book is available in full view, you can click the 'Plain text' button in the toolbar." Unfortunately the only way I've found to download the full text of a public domain book from Google is to flip through the book a page at a time, copying the text to your clipboard. There are roughly 2-3 million public domain books in Google Books. On Sat, Jun 20, 2009 at 10:10 AM, Samuel Klein <meta.sj(a)gmail.com> wrote:

Not likely. I've been banned from Google's regular search at least a dozen times during semi-frenetic search sprees in which I was identified as a bot. There is no doubt that if you try to automate it you will be quickly shot down. On Sat, Jun 20, 2009 at 12:02 PM, Platonides <Platonides(a)gmail.com> wrote:

So the bot just has to run at human speeds so it does not get banned, it still won't get tired or make unpredictable mistakes. And you can run it from different IPs to parallelize. --Falcorian On Sat, Jun 20, 2009 at 11:04 AM, Brian <Brian.Mingus(a)colorado.edu> wrote:

For some reason, I am reminded of a Supreme Court case about the information in telephone directories. Maybe because of the insanity of trying to put public domain material under copyright. ________________________________ From: Brian &lt;Brian.Mingus(a)colorado.edu&gt; To: Wikimedia Foundation Mailing List &lt;foundation-l(a)lists.wikimedia.org&gt; Sent: Saturday, June 20, 2009 11:47:28 AM Subject: Re: [Foundation-l] Info/Law blog: Using Wikisource as an Alternative Open Access Repository for Legal Scholarship That is against the law. It violates Google's ToS. I'm mostly complaining that Google is being Very Evil. There is nothing we can do about it except complain to them. Which I don't know how to do - they apparently believe that the plain text versions of their books are akin to their intellectual property and are unwilling to give them away. On Sat, Jun 20, 2009 at 12:34 PM, Falcorian < alex.public.account+WikimediaMailingList@gmail.com<alex.public.account%2BWikimediaMailingList@gmail.com> > So the bot just has to run at human speeds so it does not get banned, it > still won't get tired or make unpredictable mistakes. And you can run it > from different IPs to parallelize. > > --Falcorian > > On Sat, Jun 20, 2009 at 11:04 AM, Brian &lt;Brian.Mingus(a)colorado.edu> > > Not likely. I've been banned from Google's regular search at least a > dozen > > times during semi-frenetic search sprees in which I was identified as a > > bot. > > There is no doubt that if you try to automate it you will be quickly shot > > down. > > > > On Sat, Jun 20, 2009 at 12:02 PM, Platonides &lt;Platonides(a)gmail.com&gt; > > > > > Brian wrote: > > > > Unfortunately the only way I've found to download the full text of a > > > public > > > > domain book from Google is to flip through the book a page at a time, > > > > copying the text to your clipboard. > > > > There are roughly 2-3 million public domain books in Google Books. > > > > > > That's easy to fix :) > > > > > > > > > _______________________________________________ > > > foundation-l mailing list > > > foundation-l(a)lists.wikimedia.org > > > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l > > > > > _______________________________________________ > > foundation-l mailing list > > foundation-l(a)lists.wikimedia.org > > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l > > > _______________________________________________ > foundation-l mailing list > foundation-l(a)lists.wikimedia.org > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l > _______________________________________________ foundation-l mailing list foundation-l(a)lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

On Sat, Jun 20, 2009 at 1:29 PM, Platonides &lt;Platonides(a)gmail.com&gt; wrote: 5.3 You agree not to access (or attempt to access) any of the Services by any means other than through the interface that is provided by Google, unless you have been specifically allowed to do so in a separate agreement with Google. You specifically agree not to access (or attempt to access) any of the Services through any automated means (including use of scripts or web crawlers) and shall ensure that you comply with the instructions set out in any robots.txt file present on the Services.

Wow, what's Wikipedia's policy about using a bot to scrape everything? On Sat, Jun 20, 2009 at 2:47 PM, Brian &lt;Brian.Mingus(a)colorado.edu&gt; wrote:

Evil I tell you. Evil! On Sat, Jun 20, 2009 at 7:56 PM, Ray Saintonge &lt;saintonge(a)telus.net&gt; wrote:

If a bot has a meaningful effect on server load (i.e. page requests), it falls under the category of malicious software, which is highly illegal. ________________________________ From: Ray Saintonge &lt;saintonge(a)telus.net&gt; To: Wikimedia Foundation Mailing List &lt;foundation-l(a)lists.wikimedia.org&gt; Sent: Saturday, June 20, 2009 2:35:52 PM Subject: Re: [Foundation-l] Info/Law blog: Using Wikisource as an Alternative Open Access Repository for Legal Scholarship Brian wrote: How is violating Google's ToS against the law? Sites put all sorts of meaningless garbage into these documents, and users mostly ignore them. Of course Google's evil; it's about time that people noticed that. They use their deep pockets as a way to bully other sites ... with a smile. Fortunately the U.S. does not have database protection laws like the E.U. Ideally, every PD item they host should also be hosted on an alternative site, but that's a massive undertaking, ... and they know it. Nothing requires them to be nice to the competition, such as by making it easy to copy their material. Ec _______________________________________________ foundation-l mailing list foundation-l(a)lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

A bot or bots calling up massive amounts of data at high speed can have a negative effect on a server. While I doubt the bot we use would have the power to take down a Google server, the speed of the requests and the constant number of requests will definitely be noticeable, possibly leading to unpleasant consequences. ________________________________ From: Ray Saintonge &lt;saintonge(a)telus.net&gt; To: Wikimedia Foundation Mailing List &lt;foundation-l(a)lists.wikimedia.org&gt; Sent: Saturday, June 20, 2009 5:07:44 PM Subject: Re: [Foundation-l] Info/Law blog: Using Wikisource as an Alternative Open Access Repository for Legal Scholarship Geoffrey Plourde wrote: Malicious software or overloading servers goes well beyond ignoring a ToS. Why should downloading whole books from Google have any greater effect on server load than downloading a whole book of similar length from Internet Archive? Ec _______________________________________________ foundation-l mailing list foundation-l(a)lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

On Sat, Jun 20, 2009 at 14:35, Ray Saintonge&lt;saintonge(a)telus.net&gt; wrote: The verdict in _United States v. Lori Drew_ appears to set a precedent that violating a site's Terms of Service is a violation of the Computer Fraud and Abuse Act. It's not a very strong precedent, but it's still there. -- Mark [[en:User:Carnildo]]

Дана Saturday 20 June 2009 18:29:24 Brian написа: Often, these books are available in the Million Books Project too.

2009/6/23 Samuel Klein &lt;meta.sj(a)gmail.com&gt; The dataset you need to train an OCR system to be as good as theirs is the raw images and the plain text. They aren't making it easy to get either of those things :( They have presumably improved the software in other ways as well.. WTF GOOG?

On Tue, Jun 23, 2009 at 11:44 AM, Michael Snow &lt;wikipedia(a)verizon.net&gt;wrote;wrote: I get the joke but um, I used it on purpose and which one of my arguments been "coopted" ??

Ok Shakespeare. But in plain english you appear to be saying that corporations are inherently greedy and have a tendency to be evil. Sure, but we expect more out of GOOG. This is not MSFT we are talking about. On Tue, Jun 23, 2009 at 12:13 PM, Michael Snow &lt;wikipedia(a)verizon.net&gt;wrote;wrote:

On Tue, Jun 23, 2009 at 3:58 PM, Anthony &lt;wikimail(a)inbox.org&gt; wrote: I guess even a non-profit is inherently greedy, it's just greedy for something other than money. The WMF is greedy for the spread of free knowledge. But this is off-topic. Let's take it to another list or something.

On Tue, Jun 23, 2009 at 1:09 PM, Brian &lt;Brian.Mingus(a)colorado.edu&gt; wrote: It's almost like they're trying to run a business or something.

On Sun, Jun 21, 2009 at 1:41 AM, David Gerard &lt;dgerard(a)gmail.com&gt; wrote: Tim Armstrong is a sysop on Wikisource ... :-) more below.. On Sun, Jun 21, 2009 at 4:17 PM, Ray Saintonge &lt;saintonge(a)telus.net&gt; wrote: If it was published, Wikisource accepts it. Notability is not a consideration. The only other "open" project of comparable size is [[Distributed Proofreaders]]. Here are our statistics: http://wikisource.org/wiki/Wikisource:ProofreadPage_Statistics Most of the Wikisource projects accept free translations. http://wikisource.org/wiki/WS:COORD The two English Wikisource featured translations are: http://en.wikisource.org/wiki/Balade_to_Rosemounde http://en.wikisource.org/wiki/J%27accuse (also translated into Dutch) The two biggest translation projects that I know of are: http://en.wikisource.org/wiki/Romance_of_the_Three_Kingdoms http://en.wikisource.org/wiki/Bible_(Wikisource) Another good one is http://en.wikisource.org/wiki/Max_Havelaar_(Wikisource) We also have translations of laws, usually relating to copyright. http://en.wikisource.org/wiki/Ordinance_93-027_of_30_March_1993_on_copyrigh… Tim has been doing high impact work in this area. H.R. Rep. No. 94-1476 http://blogs.law.harvard.edu/infolaw/2008/06/17/an-open-access-success-stor… U.S. Statutes at Large http://blogs.law.harvard.edu/infolaw/2008/06/02/public-records-one-jpeg-at-… http://en.wikisource.org/wiki/United_States_Statutes_at_Large In regards the USC, the majority of it is a mess, but Title 17 is a great example of where we are heading. http://en.wikisource.org/wiki/United_States_Code/Title_17 We also have transcription projects for the UK 1911 copyright act, which has influenced so many other countries. http://en.wikisource.org/wiki/Index:The_copyright_act,_1911,_annotated.djvu http://en.wikisource.org/wiki/Index:A_treatise_upon_the_law_of_copyright.dj… More can be found from our freshly minted Law index: http://en.wikisource.org/wiki/Wikisource:Law Our two featured texts are: http://en.wikisource.org/wiki/South_Africa_Act_1909 http://en.wikisource.org/wiki/ACLU_v._NSA_(District_Court_opinion) If people want to help, but dont know where to start, my recommendation is that they start proofreading the Stat. volume 1, as this is goldmine of interesting documents, and will be an excellent example of crowdsourcing of transcription. http://en.wikisource.org/wiki/Index:United_States_Statutes_at_Large/Volume_1 Enjoy, John Vandenberg