Since nobody reads mobile-l
---------- Forwarded message ---------- From: Jon Robson jdlrobson@gmail.com Date: Sat, Jun 8, 2013 at 4:17 AM Subject: [WikimediaMobile] Number crunching: Upload errors on mobile To: mobile-l mobile-l@lists.wikimedia.org
I ran some data crunching on a sample of 4429 photo uploads from mobile web. In this sample 2821 uploads succeeded and 36% (1608/4429) of attempted uploads failed. This is very high and unacceptable.
Looking closely 53% of all errors were due to problems with invalid or 'anonymous' tokens. This will occur when the client is unable to get a token using CORS from Commons due to not being logged in there. Luckily Chris Steipp and the rest of the platform team have pushed a change that should significantly reduce this error: https://gerrit.wikimedia.org/r/#/c/57662/
CentralAuth related errors (861) #################################### Anonymous token. 598 Invalid token 263
The next big offender was 'Missing filename' accounting for 22% of all upload errors. Unfortunately this is ambiguous as it could mean a variety of things - it simply means that an upload was attempted and the response didn't report the filename. I've pushed a patch to try to understand what errors we are running into: https://gerrit.wikimedia.org/r/67545
Other errors (467) ################################### Missing filename 347 This file did not pass file verification 69 Blank error message given 36 The file you submitted was empty 15
There are various other errors all listed below for your enjoyment. Some due to bad choices of name, permissions problems and attempts to upload certain file types we do not accept. The good news is we probably don't want these uploads to succeed as they hint at vandalism attempts or uploads by poorly educated users.
The server problems section is worth a look though - although a small percentage "The modification you tried to make was aborted by an extension hook 61". These errors are occurring on the following wiki projects: * sv.m.wikipedia.org * de.m.wikipedia.org * test.m.wikipedia.org * en.m.wikipedia.org * ar.m.wikipedia.org * es.m.wikipedia.org * ja.m.wikipedia.org * he.m.wikipedia.org * fr.m.wikipedia.org * nl.m.wikipedia.org Any ideas what may be causing that error?
*** Other errors:
Users uploading with bad or unclear filenames (113) #################################### "titleblacklist-custom-filename" 48 (https://commons.m.wikimedia.org/wiki/Template:Titleblacklist-custom-filename...) "titleblacklist-forbidden-edit" 37 (https://en.m.wikipedia.org/wiki/MediaWiki_talk:Titleblacklist-forbidden-edit) Filename exists 25 (I suspect they used a common filename) Unknown error: "titleblacklist-custom-double-apostrophe" 3
(Out of interest is there any API to check whether a filename will be accepted?)
Server problems (81): #################################### The modification you tried to make was aborted by an extension hook 61 Database query error 10 An internal error occurred 9 error: Internal Server Error 1
Permission based errors (69) #################################### The "autoconfirmed" right is required to edit this page 36 You have been blocked from editing 25 The "protect" right is required to edit this page 7 Unknown error: "globalblocking-ipblocked" 1
(These users should not be seeing the upload button!)
Users uploading with bad file types (17): ########################################### Filetype not permitted: MOV 8 Filetype not permitted: webp 4 Filetype not permitted: mp3 3 Filetype not permitted: xml 1 Filetype not permitted: bmp 1
_______________________________________________ Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Yuvi Panda T http://yuvi.in/blog
The server problems section is worth a look though - although a small percentage "The modification you tried to make was aborted by an extension hook 61". These errors are occurring on the following wiki projects:
- sv.m.wikipedia.org
- de.m.wikipedia.org
- test.m.wikipedia.org
- en.m.wikipedia.org
- ar.m.wikipedia.org
- es.m.wikipedia.org
- ja.m.wikipedia.org
- he.m.wikipedia.org
- fr.m.wikipedia.org
- nl.m.wikipedia.org
Any ideas what may be causing that error?
I suspect that is caused by UploadBlacklist extension, which blacklists about 23 files by their sha hash. According to the config file, there's a log at "udp://$wmfUdp2logDest/upload-blacklist", so you can probably check if that guess is right.
--bawolff
On 08.06.2013, 21:07 Brian wrote:
I suspect that is caused by UploadBlacklist extension, which blacklists about 23 files by their sha hash. According to the config file, there's a log at "udp://$wmfUdp2logDest/upload-blacklist", so you can probably check if that guess is right.
$ grep -v 'MISS' upload-blacklist.log $
Update on error numbers:
111 errors since we pushed the fix on Tuesday 2pm to avoid anonymous token problems.
Things are looking much better and come under 3 types of error Here is the new breakdown
1) Errors due to bad filenames / bad filetypes
Missing filename: Bad filename 67
This is now the biggest cause of errors. I suggest we log the file names
users are trying to upload with to identify what is going wrong. A bug is open https://bugzilla.wikimedia.org/show_bug.cgi?id=49544
There are various other errors which are probably not worth our time as they suggest vandal edits:
This file did not pass file verification 4 Unknown error: "titleblacklist-forbidden-edit" 5 Unknown error: "titleblacklist-custom-filename" 1 Missing filename: Filename exists 1 Missing filename: Duplicate archive 1 Filetype not permitted: MOV 1
The file type not permitted errors could be solved by checking the image
in preview mode correctly loaded and whether it has a width that isn't 0.
2) Errors due to tokens
Bad token name. 15
These have all occurred on en.m.wikipedia.org, ru.m.wikipedia.org and
meta.m.wikimedia.org
5 of them were on the uploads page
Investigating with Chris it seems this can happen when a user has logged in on mobile, left the page for some time and thus their login has expired. Requesting a central auth token without being logged in will cause this error. We should explore checking login status before starting the upload workflow and redirect the user to the login page.
Invalid token 5
We are caching tokens that have since expired. We should explore
invalidating tokens.
3) Other errors These bugs are all pretty mysterious and it's not clear what causes them
* The modification you tried to make was aborted by an extension hook 7 * error: 3
Note This occurs when an error happens but the error is missing an 'info'
property. e.g. the response is: { error: {} }
An internal error occurred 1
On Sat, Jun 8, 2013 at 3:49 PM, Max Semenik maxsem.wiki@gmail.com wrote:
On 08.06.2013, 21:07 Brian wrote:
I suspect that is caused by UploadBlacklist extension, which blacklists about 23 files by their sha hash. According to the config file, there's a log at "udp://$wmfUdp2logDest/upload-blacklist", so you can probably check if that guess is right.
$ grep -v 'MISS' upload-blacklist.log $
-- Best regards, Max Semenik ([[User:MaxSem]])
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
-- Jon Robson http://jonrobson.me.uk @rakugojon
Nice work! Resolving the bad filename issue should get us down to a much more acceptable error rate.
On Fri, Jun 21, 2013 at 3:19 PM, Jon Robson jdlrobson@gmail.com wrote:
Update on error numbers:
111 errors since we pushed the fix on Tuesday 2pm to avoid anonymous token problems.
Things are looking much better and come under 3 types of error Here is the new breakdown
- Errors due to bad filenames / bad filetypes
Missing filename: Bad filename 67
This is now the biggest cause of errors. I suggest we log the file names
users are trying to upload with to identify what is going wrong. A bug is open https://bugzilla.wikimedia.org/show_bug.cgi?id=49544
There are various other errors which are probably not worth our time as they suggest vandal edits:
This file did not pass file verification 4 Unknown error: "titleblacklist-forbidden-edit" 5 Unknown error: "titleblacklist-custom-filename" 1 Missing filename: Filename exists 1 Missing filename: Duplicate archive 1 Filetype not permitted: MOV 1
The file type not permitted errors could be solved by checking the image
in preview mode correctly loaded and whether it has a width that isn't 0.
- Errors due to tokens
Bad token name. 15
These have all occurred on en.m.wikipedia.org, ru.m.wikipedia.org and
meta.m.wikimedia.org
5 of them were on the uploads page
Investigating with Chris it seems this can happen when a user has logged in on mobile, left the page for some time and thus their login has expired. Requesting a central auth token without being logged in will cause this error. We should explore checking login status before starting the upload workflow and redirect the user to the login page.
Invalid token 5
We are caching tokens that have since expired. We should explore
invalidating tokens.
- Other errors
These bugs are all pretty mysterious and it's not clear what causes them
- The modification you tried to make was aborted by an extension hook 7
- error: 3
Note This occurs when an error happens but the error is missing an
'info' property. e.g. the response is: { error: {} }
An internal error occurred 1
On Sat, Jun 8, 2013 at 3:49 PM, Max Semenik maxsem.wiki@gmail.com wrote:
On 08.06.2013, 21:07 Brian wrote:
I suspect that is caused by UploadBlacklist extension, which blacklists about 23 files by their sha hash. According to the config file, there's a log at "udp://$wmfUdp2logDest/upload-blacklist", so you can probably check if that guess is right.
$ grep -v 'MISS' upload-blacklist.log $
-- Best regards, Max Semenik ([[User:MaxSem]])
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
-- Jon Robson http://jonrobson.me.uk @rakugojon
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Update: I did some more exploring and discovered that very long filenames can cause the bad filename issue. I've added a margin of error to our file length chooser algorithm: https://gerrit.wikimedia.org/r/#/c/70223/
Whilst there I also added protection against uploading of bad file types, filenames with double apostrophes and better logging for bad filenames.
I suspect these changes will drastically cut the upload error rate on mobile web.
On Mon, Jun 24, 2013 at 11:18 AM, Arthur Richards arichards@wikimedia.org wrote:
Nice work! Resolving the bad filename issue should get us down to a much more acceptable error rate.
On Fri, Jun 21, 2013 at 3:19 PM, Jon Robson jdlrobson@gmail.com wrote:
Update on error numbers:
111 errors since we pushed the fix on Tuesday 2pm to avoid anonymous token problems.
Things are looking much better and come under 3 types of error Here is the new breakdown
- Errors due to bad filenames / bad filetypes
Missing filename: Bad filename 67
This is now the biggest cause of errors. I suggest we log the file names users are trying to upload with to identify what is going wrong. A bug is open https://bugzilla.wikimedia.org/show_bug.cgi?id=49544
There are various other errors which are probably not worth our time as they suggest vandal edits:
This file did not pass file verification 4 Unknown error: "titleblacklist-forbidden-edit" 5 Unknown error: "titleblacklist-custom-filename" 1 Missing filename: Filename exists 1 Missing filename: Duplicate archive 1 Filetype not permitted: MOV 1
The file type not permitted errors could be solved by checking the image in preview mode correctly loaded and whether it has a width that isn't 0.
- Errors due to tokens
Bad token name. 15
These have all occurred on en.m.wikipedia.org, ru.m.wikipedia.org and meta.m.wikimedia.org 5 of them were on the uploads page
Investigating with Chris it seems this can happen when a user has logged in on mobile, left the page for some time and thus their login has expired. Requesting a central auth token without being logged in will cause this error. We should explore checking login status before starting the upload workflow and redirect the user to the login page.
Invalid token 5
We are caching tokens that have since expired. We should explore invalidating tokens.
- Other errors
These bugs are all pretty mysterious and it's not clear what causes them
- The modification you tried to make was aborted by an extension hook 7
- error: 3
Note This occurs when an error happens but the error is missing an 'info' property.
e.g. the response is: { error: {} }
An internal error occurred 1
On Sat, Jun 8, 2013 at 3:49 PM, Max Semenik maxsem.wiki@gmail.com wrote:
On 08.06.2013, 21:07 Brian wrote:
I suspect that is caused by UploadBlacklist extension, which blacklists about 23 files by their sha hash. According to the config file, there's a log at "udp://$wmfUdp2logDest/upload-blacklist", so you can probably check if that guess is right.
$ grep -v 'MISS' upload-blacklist.log $
-- Best regards, Max Semenik ([[User:MaxSem]])
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
-- Jon Robson http://jonrobson.me.uk @rakugojon
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687
wikitech-l@lists.wikimedia.org