On 1 May 2014 13:21, Federico Leva (Nemo) <nemowiki(a)gmail.com> wrote: Not on bugzilla yet. I'd like to know if others have a similar experience of GWT halting mid-run for these sorts of reasons. Unfortunately with no error reporting back from partial runs, diagnosis becomes quite hard, even working out which file your run failed on! Charsets are quite a tricky thing to debug as this sort of error can boil down to the local file encoding (my xml should be 'utf-8', I am working on an OSX system, and my xml header is correct) but for all I know my use of JEdit could be creating discrepencies. Fae

On 01/05/2014, Fæ <faewik(a)gmail.com> wrote: ... After downloading the linked tiff file in the previous example, that may be a tangent, as the NYPL might be failing to put the right mime data on some of their tiffs. I doubt I'll ever fix that. Instead, a good example of characters giving a problem is the file at [1]. This caused the GWT run to halt but was successfully loaded once I changed the "Æ" (ae ligature) character in Ægean to a simple "A". The only cause of this failure must have been the character, which is allowed in the mediawiki software. Links 1. https://commons.wikimedia.org/wiki/File:A_new_map_of_the_islands_of_the_Age… Fae -- faewik(a)gmail.com https://commons.wikimedia.org/wiki/User:Fae

characters ---------- i have a test xml i use to test titles and added the characters you mentioned. i had no problem uploading the test xml file. here are 2 results that seem to indicate that there should not be an issue with the characters: http://commons.wikimedia.beta.wmflabs.org/wiki/File:The_%22King%E2%80%99s_o… http://commons.wikimedia.beta.wmflabs.org/wiki/File:Dice_players_-_Lo_L%C3%… example record -------------- i tested the example record locally and after about 2 minutes i got the message: The file you submitted was too large. original URL: http://link.nypl.org/2Qqj_oLvSbWRwPxtB1rq_wZ evaluated URL: http://link.nypl.org/2Qqj_oLvSbWRwPxtB1rq_wZ my wiki was set to a limit of 100mb, so i up’d it to 1000mb. i also switched to the new preview branch i have in gerrit, https://gerrit.wikimedia.org/r/#/c/127839/, for bug https://bugzilla.wikimedia.org/show_bug.cgi?id=63864, which no longer downloads an image to the wiki during the preview step. instead it downloads all mediafiles in a background job. the job successfully completed after 3 minutes and the image was viewable in my local wiki. i also took a look at our wikitech instance and saw that you had uploaded the image there without issue. i also repeated the uploaded but got the message: “This file did not pass file verification.” this seems to have been thrown by UploadBase.php, so i'd have to look further into that issue. but i also suspect that commons may have just timed out on the download of the image in the preview step. this type of error seems similar to bug 63864. i just need someone to +2 the patch i made so that we can test the new preview step on the beta cluster. with kind regards, dan On May 1, 2014, at 18:35 , Federico Leva (Nemo) <nemowiki(a)gmail.com> wrote:

Could you include what the binary code for your ae was if possible (on unix computers, possibly also mac, the hd or hexdump command can tell you this) or just attach the xml in question to a bug (since there is a possibility that your email client might change the character)? The encoding should be utf8 with NFC, but even if its not quite correct gw/mw should convert it. --bawolff On May 2, 2014 3:42 AM, "Fæ" <faewik(a)gmail.com> wrote: the way I'm creating my xml (Python generated, then edited in JEdit for any tweaks, should be 'utf-8') so I'll continue plugging at it. why. mentioned. i had no problem uploading the test xml file. here are 2 results that seem to indicate that there should not be an issue with the characters: http://commons.wikimedia.beta.wmflabs.org/wiki/File:The_%22King%E2%80%99s_o… http://commons.wikimedia.beta.wmflabs.org/wiki/File:Dice_players_-_Lo_L%C3%… >> example record >> -------------- >> i tested the example record locally and after about 2 minutes i got the message: http://link.nypl.org/2Qqj_oLvSbWRwPxtB1rq_wZ evaluated URL: http://link.nypl.org/2Qqj_oLvSbWRwPxtB1rq_wZ https://gerrit.wikimedia.org/r/#/c/127839/, for bug https://bugzilla.wikimedia.org/show_bug.cgi?id=63864, which no longer downloads an image to the wiki during the preview step. instead it downloads all mediafiles in a background job. viewable in my local wiki. uploaded the image there without issue. i also repeated the uploaded but got the message: further into that issue. but i also suspect that commons may have just timed out on the download of the image in the preview step. this type of error seems similar to bug 63864. i just need someone to +2 the patch i made so that we can test the new preview step on the beta cluster. >> with kind regards, >> dan >> On May 1, 2014, at 18:35 , Federico Leva (Nemo) <nemowiki(a)gmail.com> wrote: https://commons.wikimedia.org/wiki/File:A_new_map_of_the_islands_of_the_Age… bug report. Please file. :) >> > >> > Nemo >> > >> > >> > _______________________________________________ >> > Glamtools mailing list >> > Glamtools(a)lists.wikimedia.org >> > https://lists.wikimedia.org/mailman/listinfo/glamtools >> _______________________________________________ >> Glamtools mailing list >> Glamtools(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/glamtools > > > > > -- > faewik(a)gmail.com https://commons.wikimedia.org/wiki/User:Fae > Personal and confidential, please do not circulate or re-quote. > > _______________________________________________ > Glamtools mailing list > Glamtools(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/glamtools >

On 2 May 2014 08:00, Brian Wolff <bawolff(a)gmail.com> wrote: ... ... SUMMARY Not a bug. Most headaches with the NYPL upload seem to be due to the source data. ANALYSIS Based on more tranches of uploads from NYPL and several days of playing with format problems, I believe this was a source site problem rather than a bug with GWT. Here's what I know based on : * My xml is created from Python correctly encoded using UTF-8. * Most of the metadata is scraped from webpages defined as "iso-8859-1" i.e. the latin-1 charset. The (key needed) API is limited, I only use it to find a direct link to the full tiff, which is mysteriously avoided in the fully open website. * Encoding of characters like "e acute" may read both correctly and incorrectly in the resulting xml. The incorrect forms exist in at least two variations for this character, however my use of encode('Windows-1252') on source fields may be part of this, I'm using this based on earlier trial and error of multiple methods. The number of records with errors is now relatively small. * Reading the file as UTF-8 in JEdit may include hidden characters, making hand-correction, tricky. These behaviours indicate to me that the NYPL source site has inconsistently encoded its web pages. If I decode or encode I get unpredictable results. At the moment, my shortest workflow seems to be to let Python create the xml as a UTF file without any more character encoding and then fix any oddities by hand. As a "more advanced" user of GWT, the fact that I am spending so much time in xml preparation should be of concern for how much we say in our user guides for the tool about xml checking, testing and preparation. I have little doubt that similar encoding problems (especially for multilanguage or ancient texts) will continue to dog some of our users. Other xml valitation issues and on-wiki conventions with regard to html encoding, hidden text etc. would be worth the user community expanding on the manual help pages; possibly growing the trouble-shooting guide as a separate document. PS with regard to Dan's timeout point, I am unsure how much this is affecting my uploads. I did have to compile an xml file of re-uploads after many were mysteriously skipped, but the second time around they appear to have all been uploaded so this performance related issue might be entirely dependant on the stress WMF's servers happen to be under. My most significant reason for skipped images is because of NYPL missing catalogue pages, however these are discovered when I create the xml file. This is something I am deferring until after uploads are complete; in practice I might not spend time investigating it as the NYPL is moving to a new website scheme, so it might make sense to park new uploads for a year and let their system stabilize. Links * Example NYPL source file with non-ascii characters: http://digitalgallery.nypl.org/nypldigital/dgkeysearchdetail.cfm?imageID=12… * API http://api.repo.nypl.org/ Fae -- faewik(a)gmail.com https://commons.wikimedia.org/wiki/User:Fae