Wikisource-l

wikisource-l@lists.wikimedia.org

4 participants
1318 discussions

Re: [Wikisource-l] Wikisource-l Digest, Vol 43, Issue 1
by Samuel Klein 22 Apr '10

22 Apr '10

A quick comment on Lars's last post: I spoke at the Museums and the Web conference last Tuesday in Denver, and someone from the Library of Congress came. They aren't a Museum, but they are working on a small-scale wikipedia project to help illustrate articles in WP with deep citations into some of their more interesting public domain holdings (which are a bit museum-like). She mentioned that she was most interested in pursuing a further wikisource project, as they have many unique works (journals, manuscripts and the like) which are only available in their original language -- say, a journal of a French botanist -- and which deserve to be translated into other languages. They would like to help publish digital scans and existing native-language text (cleaned up OCR) to wikisource, in the hopes that translations can be made, into English and other languages. They are interested in identifying the 'most interesting works' in their untranslated collections, and happy to have some discussion about what make works interesting. One of the things I like about this idea is that as curators of one of the world's largest international libraries they have a broad sense of 'notability' and interest in a wikisource sense which we currently lack on the Project... so this could help drive style guide improvements as well. Thoughts? If anyone is specifically interested in this collaboration, let me know and I'll put you in touch with the organizer. Of course once this is more than a pipe dream there will be a public project page... but early interest now could help frame the initial proposal. SJ On Wed, Apr 21, 2010 at 8:56 AM, <wikisource-l-request(a)lists.wikimedia.org> wrote: > Send Wikisource-l mailing list submissions to > wikisource-l(a)lists.wikimedia.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.wikimedia.org/mailman/listinfo/wikisource-l > or, via email, send a message with subject or body 'help' to > wikisource-l-request(a)lists.wikimedia.org > > You can reach the person managing the list at > wikisource-l-owner(a)lists.wikimedia.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Wikisource-l digest..." > > > Today's Topics: > > 1. Re: PDF/Djvu to Index (ThomasV) > 2. Re: PDF/Djvu to Index (Cecil) > 3. Strategic Planning Office Hours (Philippe Beaudette) > 4. Experience, funding, outreach (Lars Aronsson) > 5. Re: Experience, funding, outreach (Sydney Poore) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 13 Apr 2010 14:56:38 +0200 > From: ThomasV <thomasV1(a)gmx.de> > Subject: Re: [Wikisource-l] PDF/Djvu to Index > To: "discussion list for Wikisource, the free library" > <wikisource-l(a)lists.wikimedia.org> > Message-ID: <4BC46A06.3070506(a)gmx.de> > Content-Type: text/plain; charset=ISO-8859-1 > > Lars Aronsson a ?crit : >> ThomasV wrote: >> >>> the problem is that djvu pages on common >>> do not have a parsable format. >>> >> >> Many of them are. For example, >> http://commons.wikimedia.org/wiki/File:Swedish_patent_14_Mj%C3%B6lqvarn.pdf >> contains: >> >> {{Information >> |Description=Swedish patent 14: Mj?lqvarn >> |Source=Digitized by [http://nordiskapatent.se/ NordiskaPatent.se] >> |Date=January 23, 1885 >> |Author=R. Setz, J. Schweiter, Clus, Switzerland >> |Permission= >> |other_versions= >> }} >> >> This looks very formalized and parsable to me. >> I filled it in when I uploaded the file to >> Commons, and exactly the same fields need to be >> filled in manually in the newly created Index page. >> >> Maybe I should design a tool or bot that asks for >> these fields once, and then uploads the file and >> creates the Index page, based on that information. >> And my question was: Has anybody already done that? >> >> > not to my knowledge. > it is possible to request the text with the api, why dont you try it. > >> >>> What you describe is _already_ implemented : >>> when a page is created, its text is extracted >>> from the text layer of the corresponding djvu or pdf. >>> All you need to do is create djvu files with a proper text layer. >>> >> >> You are correct, it does indeed work, but only >> after I action=purge the PDF file on Commons. >> It never worked for me on the first try, >> without any purge. And I was misled by an >> earlier bug where action=purge didn't help, >> so it took me a while before I tested this. >> >> So why is the purge necessary? If OCR text >> extraction ever fails, why is this not detected >> and automatically retried? >> > purge is necessary only for files that were uploaded previously, when > text extraction was not performed. Note that text layer extraction for > pdf files is new. > >> When I try to create >> http://sv.wikisource.org/wiki/Sida:Swedish_patent_14_Mj%C3%B6lqvarn.pdf/1 >> there is a character encoding error in the OCR text. >> It looks as if the PDF contains 8-bit data, which >> is loaded into the UTF-8 form without conversion. >> Cut-and-paste from Acrobat Reader works fine. >> > yes there is a conversion problem with pdf; it works better with djvu. > > > > > > ------------------------------ > > Message: 2 > Date: Tue, 13 Apr 2010 16:43:47 +0300 > From: Cecil <cecilatwp(a)gmail.com> > Subject: Re: [Wikisource-l] PDF/Djvu to Index > To: "discussion list for Wikisource, the free library" > <wikisource-l(a)lists.wikimedia.org> > Message-ID: > <y2ucec8cb611004130643jfbc6bd1cq6b4f561c5c8e3929(a)mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > 2010/4/13 ThomasV <thomasV1(a)gmx.de> > >> Lars Aronsson a ?crit : >> > ThomasV wrote: >> > >> >> the problem is that djvu pages on common >> >> do not have a parsable format. >> >> >> > >> > Many of them are. For example, >> > >> http://commons.wikimedia.org/wiki/File:Swedish_patent_14_Mj%C3%B6lqvarn.pdf >> > contains: >> > >> > {{Information >> > |Description=Swedish patent 14: Mj?lqvarn >> > |Source=Digitized by [http://nordiskapatent.se/ NordiskaPatent.se] >> > |Date=January 23, 1885 >> > |Author=R. Setz, J. Schweiter, Clus, Switzerland >> > |Permission= >> > |other_versions= >> > }} >> > >> > This looks very formalized and parsable to me. >> > I filled it in when I uploaded the file to >> > Commons, and exactly the same fields need to be >> > filled in manually in the newly created Index page. >> > >> > Maybe I should design a tool or bot that asks for >> > these fields once, and then uploads the file and >> > creates the Index page, based on that information. >> > And my question was: Has anybody already done that? >> > >> > >> not to my knowledge. >> it is possible to request the text with the api, why dont you try it. >> >> > > Two problems. > > 1. I don't think all projects are using this template as it does not really > fit for books: At least German Wikisource has special templates for books, > single pages, djvu-Files, pdf-Files and so on. It uses mostly those for its > Commons-uploads and not the non-specific information-template as this > template does not have the parameters needed to describe book data (author, > publisher, place of publishing, year of first publishing, publishing > version, year of publishing of this version, ...). And AFAIK de.WS is not > the only project which uses specialized templates for its Commons-uploads. > > 2. I'm not sure about this but I think the index-file has the same fields in > all projects which use the extension. That would mean that the > Information-template does not contain the correct data for filling the > index-page. At least the index-file on de.WS has separate fields for author > and publisher and year of publishing and place of publishing and we usually > also link locally to the author and the title-page. So at least from your > example up there only one parameter (the source) is really useable (and at > least I usually linked to the Commons-file in the source-parameter on the > WS-indexPage). I'm not sure how much time the parse-request would need but > it does not look really worth the time (both programming it and later using > it) considering its useable return values. > > IMO you could create a lot of index-files in the time you spend figuring out > if it is possible to extract, parse and interpret the data from Commons. > There are too many uploads which do not use any template, other templates or > while using this template are filled out in an unuseable way (as everybody > has a little bit a different style even when using templates), and even then > it lacks half the needed information while the rest still needs formatting. > The benefit is quite small compared to the amount of work it requires. > > But hey, if you have spare time it would be still interesting to know if you > can get the data in a way that would not slow down users with not-so-fast > internet-connections. > > Cecil >

1 0

OCR requests
by Lars Aronsson 22 Apr '10

22 Apr '10

The Swedish Wikisource is copying scanned books from various sources. You typically find a PDF or DJVU file, containing both scanned images and raw OCR text, that you upload to Commons, create an Index: page with the <pagelist/> tag. Some of these books have pretty miserable OCR text, perhaps because the Norwegian National library scanned a Swedish book with their OCR software set to Norwegian. Somebody with an OCR program needs to run a new OCR on these images. Fortunately, it is quite easy to feed the PDF or DJVU file into an OCR program such as Finereader, and use a bot to update the pages. We now have one user on sv.wikisource doing this. For these Index: pages, I created a category:OCR-kö (meaning: queue of OCR requests). When trying to interwiki link, I found a similar category on de.wikisource, but similar categories on fr, en, and pt had been removed. What's the story behind that? Don't you need OCR requests in these languages? The comment on the English page mentions an OCR robot on the toolserver. Really? Exist: http://de.wikisource.org/wiki/Kategorie:OCR-Anfragen http://sv.wikisource.org/wiki/Kategori:OCR-k%C3%B6 Have been removed in June 2009: http://en.wikisource.org/wiki/Category:OCR_Requests http://fr.wikisource.org/wiki/Cat%C3%A9gorie:Demandes_d%27OCR http://pt.wikisource.org/wiki/Categoria:!Pedidos_de_OCR -- Lars Aronsson (lars(a)aronsson.se) Aronsson Datateknik - http://aronsson.se

2 1

Experience, funding, outreach
by Lars Aronsson 21 Apr '10

21 Apr '10

At the Wikimedia chapter meeting in Berlin last week, Wikisource was mentioned as an interesting project in several different settings. I know a lot of interesting projects and attempts are being tried in various languages of Wikisource, but perhaps there isn't enough coordination and exchange of ideas and experience between all volunteers. How could we improve this? I personally think this mailing list is the first place to start. We could all write short notices of any new idea or project that we are undertaking. Then we should probably get together at a session during the Wikimania conference in Gdansk this summer. The Wikimedia "chapters" are national or regional membership associations that provide means to go beyond the ordinary project volunteer communities, for example when expenses need to be covered for travel or equipment, or when contracts need to be signed. One example is that Wikimedia France recently signed a deal with the Bibliothèque nationale de France to provide access to scanned images of books, that can be proofread in fr.wikisource.org. Deals of this kind fit in with a larger pattern, where chapters seek collaboration with galleries, libraries, archives and museums (= GLAM), hoping that they will contribute free images to Wikimedia Commons and Wikipedia. Or where museums will allow wikipedians to take photos of their collections. Other chapters are buying scanners for volunteers to use. But perhaps digital cameras are more useful than scanners these days. How many know how to use them correctly? Maybe we need workshops. Many of the chapters are now growing fast and are quite successful at fundraising. This creates an interesting challenge to fund projects that really make a difference. The chapters need good projects to fund, that they can show off to donors at the coming fundraiser in the fall 2010/winter 2011. I think Wikisource has a lot of potential for supplying chapters with good projects to fund. -- Lars Aronsson (lars(a)aronsson.se) Wikimedia Sverige - stöd fri kunskap - http://wikimedia.se/

2 1

Strategic Planning Office Hours
by Philippe Beaudette 19 Apr '10

19 Apr '10

Hi everyone, The next strategic planning office hours are: Tuesday, 6 April, from 20:00-21:00 UTC, which is: -Tuesday (1-2pm PDT) -Tuesday (4-5pm EDT) Office hours will be a great opportunity to discuss the work that's happened as well as the work to come. As always, you can access the chat by going to https://webchat.freenode.net and filling in a username and the channel name (#wikimedia-strategy). You may be prompted to click through a security warning. It's fine. More details at: http://strategy.wikimedia.org/wiki/IRC_office_hours Thanks! Hope to see many of you there. ____________________ Philippe Beaudette Facilitator, Strategy Project Wikimedia Foundation philippe(a)wikimedia.org Imagine a world in which every human being can freely share in the sum of all knowledge. Help us make it a reality! http://wikimediafoundation.org/wiki/Donate

1 1

PDF/Djvu to Index
by Lars Aronsson 13 Apr '10

13 Apr '10

It is increasingly common to add books to Wikisource by finding a PDF or Djvu file, uploading it to Commons, and then to create an Index: page on Wikisource for proofreading. But this would be much easier if: 1) The fields (author, title, etc.) of the Index page were filled in from the data already given on Commons. (Yes, those could be wrong or need additional care, but this could always be edited afterwards, if initial values are fetched from Commons.) 2) The <pagelist/> tag was already in the "pages" box. 3) All pages were created in automatically with the OCR text from Commons, instead of leaving a long list of red links. (This would require the text for each page to be extracted, something that pdftotext can do in seconds, but Commons takes weeks to do.) Could this be automated? Is there already some tool or bot that does this? -- Lars Aronsson (lars(a)aronsson.se) Aronsson Datateknik - http://aronsson.se

3 4

Re: [Wikisource-l] Are the Page Views statistics fair enough ?
by Michel Morin 06 Apr '10

06 Apr '10

Ok for this one, then. But many others on other wikisources are, I guess, unfair. Does any admin or developper have the rights to modify this stuff on each domain ? Regards, Syagrius De : Michael Jörgens Envoyés : 15.03.10 18:18 À : discussion list for Wikisource, the free library Objet : Re: [Wikisource-l] Are the Page Views statistics fair enough ? {{externesBild}} is quite understandable. In german language wikisource it is mandatory that the scan of the page is public available and linked, I think it is something like Special:Filepath/XX. A lot of old books a scanned by universities und we link to the pages there, but our main source is commons for the scans- greetings 2010/3/15 <syagrius(a)gmx.fr> Since the wikisource sub-domains are now classified by page views count, I guess it should at least be fair. If we look to the statistics for December 2009, we see that an important part of the English wikisource traffic (http://stats.grok.se/en.s/ http://stats.grok.se/en.s/ ) comes from "Special:AutoLogin", of the Russian one (http://stats.grok.se/ru.s/ http://stats.grok.se/ru.s/ ) from "Special:Filepath/XXpng", of the German one (http://stats.grok.se/de.s/ http://stats.grok.se/de.s/ ) from "{{{EXTERNESBILD}}}" (I don't know what it is) and some "png", and of the Spanish one (http://stats.grok.se/es.s/ http://stats.grok.se/es.s/ ) from "Special:Filepath/XXpng", etc. There was the same problem on French wikisource some months ago, and it was fortunately corrected. Could anyone correct all this on every sub-domain ? Regards, Syagrius _______________________________________________ Wikisource-l mailing list Wikisource-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikisource-l https://lists.wikimedia.org/mailman/listinfo/wikisource-l

1 0

Which scanner for books?
by Marie-Lan Tay Pamart 02 Apr '10

02 Apr '10

Hi everyone, I hope I'm not off topic on this mailing-list. I'm looking for a scanner to scan books for Wikisource. I'm rather confused by the quantity of products on the market, plus many reviews focus on film scanners. Also, I need the scanner to be Mac compatible. Would you have some advance ? Thanks in advance. -- Marie-Lan / Jastrow

3 2

Are the Page Views statistics fair enough ?
by syagrius＠gmx.fr 15 Mar '10

15 Mar '10

Since the wikisource sub-domains are now classified by page views count, I guess it should at least be fair. If we look to the statistics for December 2009, we see that an important part of the English wikisource traffic (http://stats.grok.se/en.s/) comes from "Special:AutoLogin", of the Russian one (http://stats.grok.se/ru.s/) from "Special:Filepath/XXpng", of the German one (http://stats.grok.se/de.s/) from "{{{EXTERNESBILD}}}" (I don't know what it is) and some "png", and of the Spanish one (http://stats.grok.se/es.s/) from "Special:Filepath/XXpng", etc. There was the same problem on French wikisource some months ago, and it was fortunately corrected. Could anyone correct all this on every sub-domain ? Regards, Syagrius

2 1

Strategic Planning Office hours
by Philippe Beaudette 08 Mar '10

08 Mar '10

Hi everyone, The next strategic planning office hours are: Tuesday from 20:00-21:00 UTC, which is: Tuesday, 12-1pm PST Tuesday, 3pm-4pm EST There has been a lot of tremendous work on the strategy wiki the past few months, and Task Forces are finishing up their work. Office hours will be a great opportunity to discuss the work that's happened as well as the work to come. As always, you can access the chat by going to https://webchat.freenode.net and filling in a username and the channel name (#wikimedia-strategy). You may be prompted to click through a security warning. It's fine. More details at: http://strategy.wikimedia.org/wiki/IRC_office_hours Thanks! Hope to see many of you there. ____________________ Philippe Beaudette Facilitator, Strategy Project Wikimedia Foundation philippe(a)wikimedia.org mobile: 918 200-WIKI (9454) Imagine a world in which every human being can freely share in the sum of all knowledge. Help us make it a reality! http://wikimediafoundation.org/wiki/Donate

1 5

The name "Old Wikisource"
by Milos Rancic 01 Mar '10

01 Mar '10

(Please, continue to discuss at foundation-l.) I think that the term "Old Wikisource" and wiki abbreviation "oldwikisource" is really bad for the purpose of Wikisource (hosting "the rest" of material). Something like "Multilingual Wikisource" would be better (or whatever). So, may I ask fold from Wikisource to find a better name, and fill the but at Bugzilla for changing the abbreviation?

1 0

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Wikisource-l