Hello, I'm just wondering, would it be feasible to convert wiki text (without OCR markup) back into OCR markup? A script might strip or convert markup, diff the original OCR text with the wiki text to determine what goes where, and generate the markup from scratch. You could thus cleanly convert from OCR markup to wiki markup and back without unreadable OCR markup on the wiki, and this could also be used to provide some other very useful features (I would love to accurately diff an entire Wikisource text with OCR scans of different printed documents, for example). -- Yours cordially, Jesse Martin (Pathoschild)

That's a good point. How about a much cleaner syntax that can be used to generate the OCR markup? With your example text: {{ocr line| The first experiments were made on the absorption of carbonic }} {{ocr line| acid gas by water: and here a singular disagreement was observed }} {{ocr line| in the first trials made under exactly the same circumstances. It }} This is much easier to read, you know where the line breaks go, and it's immediately clear even to someone stumbling across the text that we're specifically keeping track of lines (so they don't helpfully remove unneeded line breaks). Since single line breaks are ignored by MediaWiki, we can just use the same line width so the template syntax lines up for easier ignoring. -- Yours cordially, Jesse Martin (Pathoschild)

Hi Gregory and everyone, On Jan 22, 2008 11:22 AM, Gregory Maxwell &lt;gmaxwell(a)gmail.com&gt; wrote: I dont understand why the HTML output needs to have the DJVU markers; it could be in the raw text. Would it be acceptable to have one line per printed line, and hidden comments as required. i.e. --- The first experiments were made on the absorption of carbonic <!-- DJVU position: 551-4202-2666-4278-1 --> acid gas by water: and here a singular disagreement was observed <!-- DJVU position: ... --> in the first trials made under exactly the same circumstances. It <!-- DJVU position: ... --> --- How will words that are broken across two lines be handled ? I understand that these DJVU files will probably have a lot of corrections initially. Are you planning on updating the DJVU file on commons incrementally, or after the entire DJVU has been proof-read ? -- John

--- Ray Saintonge &lt;saintonge(a)telus.net&gt; wrote: I am agreeing with EC here. I think you are trying to do far too much with the same piece of text. Perfectly readable/editable wikimarkup and exactly macthing OCR text are not possible with the same text. I suggest you find a way to hack having the text existing twice in the proofreading page. Something like below: <!-- Here is text with OCR breaks and hyphens which matches the printed page--> Here is the wikimarkup text that is trancluded to the WS page Of course this means both sets of text need to be proofread, but I think a script should be able highlight all the differences between them making it simple to proofread one from the other. If you really want to have only one version of the text, it will have to have the exactness of OCR sacrificed. People will always go through the markup "fixing" the hyphens. Birgitte SB ____________________________________________________________________________________ Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

On Jan 23, 2008 2:26 AM, Gregory Maxwell &lt;gmaxwell(a)gmail.com&gt; wrote: The hyphen is a difficult problem, but it doesnt need to be a deal breaker. Bidirectional djvu/pdf <-> wiki should be our goal, if we are to migrate to having all content backed by images (which is a strict policy on the German Wikisource project), but there are a few hurdles. The two big ones are: 1. hyphens This may be easily solved by replacing hyphens that appear at the end of a line with a soft-hyphen in the initial OCR output; any places where a hard-hyphens or non-breaking hyphen is required, they will be fixed in proof reading. Most browsers handle this correctly by simply discarding the soft-hyphen, and after years of waiting Firefox 3 should render this correctly < https://bugzilla.mozilla.org/show_bug.cgi?id=9101 >. Unless there are some surprises in mediawiki's handling of soft-hyphens, the wikitext would look like (in this example, &shy; could be the Unicode equivalent) <!-- any OCR sync information -->line one with a extra&shy;<!-- any OCR sync information -->ordinary size words that flow onto line two <!-- any OCR sync information --> and here is line three. i.e. original line one and two would need to be on a single line in order to prevent a newline to be emitted in the HTML, which would invalidate the soft-hyphen. If we wanted to get fancy, the dev's could enhance mediawiki so that a line ending with a Unicode soft-hyphen is not followed by a new line character in the HTML. I cant see any draw back in doing that, except for the overhead; if this is significant, it could be done in a Wikisource only extension. 2. wiki markup that isnt in the original This could be simply ignored by mandating that we don't add additional markup until after the text has been proof-read, and the changes have been fed back into the DJVU file. Any improvement on that position depends on improvements in the wikitext -> DJVU process. We keep a very close eye on Recentchanges and have revision patrolling for changes by non-admins, so any changes that may effect the ability to slurp improvements back into the DJVU can be managed. -- John

Ryan Dabler wrote: We would not want to change the text, but we might want to change some of it into wikilinks. Ec

Birgitte SB wrote: The whole proposal seems to come into the realm of biting off more than we can chew. I can give ThomasV's approach to having all material backed up by page scans full marks for what it sets out to do, but that still doesn't change the fact that some editors still find it more convenient to sub-optimally upload entire books from Project Gutenberg with little more additional effort than breaking off chapters into separate pages and adding headers. Unless we can get real people to do tedious but relatively non-technical tasks such as proofreading, how can we ever convince them to remain consistent with technical tasks whose benefits are far fom obvious. Eighteenth century scientific texts may have done well with only a single printing, but more popular works that had multiple editions present a challenge unless we can declare a particular printing to be canonical. The best printing for this may not be easily or cheaply available. As an example, I have an alomost complete set of the Ticknor and Fields version of the works of Thomas De Quincey. In the course of putting this together I ended up with apparently duplicate volumes. In the case of the second volume of the "Theological Essays" I have both an 1854 and an 1864 printing. The 1854 edition goes to page 276 and the 1864 edition to page 315. The later edition adds an essay missing from the earlier. The first three lines of page 71 of the 1864 printing from "Toilette of the Hebrew Lady" and ending a paragraph read "the precious stones; and at other times, the pearls were strung two and two, and their beautiful white- ness relieved by the interposition of red coral." In the 1854 printing the same text appears as lines 27-9 of page 69, except that "whiteness" now appears fully on the middle line without hyphenation. Footnotes that were at the end of an essay in 1854 are moved to the proper page in 1864. At one time, if a second printing was needed, it was easier and cheaper to reset the type, with all the attedent errors that one might imagine. Labour was cheap, and manufactured type very expensive. Ec

On Jan 27, 2008 9:55 AM, Lars Aronsson &lt;lars(a)aronsson.se&gt; wrote: The software that Gregory Maxwell is planning to use is the same that is used for Google Books. It is open source. The page scans will be uploaded to Commons as DJVU files, which are huge, and we dont really want regular updates to them. I think the way this would be handled is as separate images. e.g. If I cleaned up page 1 of [[Image:35 Sonnets by Fernando Pessoa.djvu]], I save it as [[Image:35 Sonnets by Fernando Pessoa.djvu-page1.jpg]], and I update [[Index:35 Sonnets]] to use the standalone image instead of the DJVU. Once completed, any standaline images would then be used to rebuild the DJVU file. Finding the location of a given text on an image is a novel idea. It is an interesting project that might even be suitable as a research project for a post-grad. If I understand correctly, you are suggesting that Greg uploads the DJVU without a text layer, and we all use whatever means we have available to build create the text, and then we feed the proofread text into the DJVU once complete (using vaporware software? :-) ) This has the distinct advantage of allowing the images to be improved as well as the text. We may even be able to push an improved DJVU file onto the Commons front page as a featured image. As crazy as it sounds, it is quite sane. OCR software will improve over the next year, and we want to be taking advantage of those improvements as we progress through the volumes. ThomasV has already set up the framework for user requested bot scans; we may need to extend this to handle different configurations to suit each DJVU file, so that the OCR software can "learn" as it progresses. Also, distributing the low quality OCR text in the DJVU file initially results in a many potential contributors not joining the ranks because the OCR text is "good enough". By not including the OCR text, it will encourage people to work with Wikisource to finish each volume. -- John