On Sun, May 11, 2014 at 5:04 AM, Faidon Liambotis <faidon@wikimedia.org> wrote:
The discussion between WMF multimedia team<->Commons/GLAM communities
has happened in a few places (various project/user talkpages,
glamtools@, personal emails) and to be honest I've lost track of it, so
I'm not sure what the outcome was.

I think the short-term outcome was to throttle GWToolset until there is a better fix. There is a patch pending to do that:
https://gerrit.wikimedia.org/r/#/c/132111/
https://gerrit.wikimedia.org/r/#/c/132112/

I described the thinking behind the limits in this mail and the followups:
http://thread.gmane.org/gmane.org.wikimedia.glamtools/24/focus=104
tl;dr it tries to limit the GWToolset-uploaded thumbnails appearing in Special:* at one time to 10% (5 with default settings), based on the total upload rate in the slowest hour of an average day. That's about one image per two minutes.

This is not live yet; Fae has apparently been running the upload with three threads (as a side note, GWToolset should probably log this), which means about one image per 20 sec (more precisely, three upload jobs are generated per minute; how the job queue picks them is not controlled).

(I haven't been paying the issue much attention because I assumed that GWToolset is inoperational anyway until the next deploy due to bugĀ 64931. Apparently I misunderstood how the job queue handles jobs that throw exceptions.)