On Sun, May 11, 2014 at 5:04 AM, Faidon Liambotis faidon@wikimedia.orgwrote:
The discussion between WMF multimedia team<->Commons/GLAM communities has happened in a few places (various project/user talkpages, glamtools@, personal emails) and to be honest I've lost track of it, so I'm not sure what the outcome was.
I think the short-term outcome was to throttle GWToolset until there is a better fix. There is a patch pending to do that: https://gerrit.wikimedia.org/r/#/c/132111/ https://gerrit.wikimedia.org/r/#/c/132112/
I described the thinking behind the limits in this mail and the followups: http://thread.gmane.org/gmane.org.wikimedia.glamtools/24/focus=104 tl;dr it tries to limit the GWToolset-uploaded thumbnails appearing in Special:* at one time to 10% (5 with default settings), based on the total upload rate in the slowest hour of an average day. That's about one image per two minutes.
This is not live yet; Fae has apparently been running the upload with three threads (as a side note, GWToolset should probably log this), which means about one image per 20 sec (more precisely, three upload jobs are generated per minute; how the job queue picks them is not controlled).
(I haven't been paying the issue much attention because I assumed that GWToolset is inoperational anyway until the next deploy due to bug 64931. Apparently I misunderstood how the job queue handles jobs that throw exceptions.)