Thanks for the detailed report, Faidon. Can you clarify something: for a given set of heavyweight thumbnails that need to be rendered, assuming the the uploads have ceased, would multiple visits of Special:NewFiles in a short timeframe multiply the saturation by the amount of HTTP requests to the same thumbnail URLs? I.e. if you request the URL of a thumbnail which is currently being generated because someone else requested it, does it make the issue worse? Second question is, how come piling on jobs doesn't just make the jobs that came last complete much later? The same kind of DoS situation could happen with someone bombarding us with HEAD requests on previously unrequested thumbnail sizes for small images, so I think that the issue isn't specific to large jobs. It's more a matter of properly queueing things up so that the imagescalers don't overload, regardless of the mix of job weight. On Mon, Apr 21, 2014 at 12:05 PM, Faidon Liambotis <faidon(a)wikimedia.org>wrote;wrote:

Well the Varnish request coalescing can't work for requests with login Cookie headers. I've seen code in our VCLs to disable it in those cases since the origin response cannot be reused for authenticated requests typically. On Tue, Apr 22, 2014 at 3:47 AM, Faidon Liambotis <faidon(a)wikimedia.org>wrote;wrote: -- -Aaron S

Dear Faidon, Emmanuel and Guiseppe, Thanks so much for investigating this issue so quickly and sharing the likely cause of the problem with us. This quarter, our team’s top priority is to address serious issues related to Upload Wizard — and this seems like a good one for us to take on. However, we are still in the process of releasing Media Viewer, which is likely to take most of our attention for the next few weeks. So we may not be able to troubleshoot it right away. But we are filling tickets about this issue, so we can hit the ground running in early may. Thanks again for your fine work, as well as for your patience and understanding. Fabrice On Apr 21, 2014, at 3:53 AM, Emmanuel Engelhart <emmanuel.engelhart(a)wikimedia.ch> wrote: _______________________________ Fabrice Florin Product Manager Wikimedia Foundation http://en.wikipedia.org/wiki/User:Fabrice_Florin_(WMF)

Fabrice, I don't see how a feature release can be of a higher priority than troubleshooting an outage but regardless: The outage's symptoms seem to have been alleviated since, but the Commons/GLAM communities are waiting for a response from us to resume their work. They've responded to our "pause" request and in turn requested our feedback at: https://commons.wikimedia.org/wiki/Commons:Batch_uploading/NYPL_Maps (see the large red banner with the stop sign at the bottom) ...which is also linked from: https://commons.wikimedia.org/wiki/User_talk:F%C3%A6#Large_file_uploads https://commons.wikimedia.org/wiki/Commons:Village_pump#Images_so_big_they_… Sadly, I don't have much to offer them, as I previously explained. I certainly wouldn't commit to anything considering your response on the matter. Could you communicate your team's priorities to Fæ and the rest of the Commons/GLAM community directly? Thanks, Faidon On Mon, Apr 21, 2014 at 08:57:39AM -0700, Fabrice Florin wrote:

I dont think vips would help this situation very much. (Also the vips patch is only half done. Its waiting on someone to do the other half. It is not waiting for review) At first glance (by which I mean what is said in this thread without doing any other investigation), it appears that the upload rate of files from gwtoolset is higher than the rate at which we can generate thumbnails that are "instantly" demanded. Obvious solution would be to slow down gwtoolset. Afaik there are currently multiple job runner processes that process gwtoolset jobs (--maxthreads). Lets reduce that to 1. If thats still too speedy, we could add a sleep() call to the job code. After that we could restart fae's batch upload, but with just the next 50 files. Watch ganglia stats like a hawk, if nothing bad happens, do the next 100, then 200, etc until we are reasonably sure that the gwtoolset upload rate is sustainable. Once we are sure, tell the users they can go wild again. --bawolff On Apr 22, 2014 2:27 PM, "Fabrice Florin" <fflorin(a)wikimedia.org> wrote: release. planning meeting and see if we can take it on right away. If we do, this could push back our release of Media Viewer in coming weeks. our team. We will respond here and onwiki, once our team has had a chance to investigate possible solutions. that others can view them without crashing the system. place over Easter weekend. the course of 4-5 hours (multiple files per minute), and then a thumbnail for all of those new files in parallel in realtime, and thus inadvertently DoSing them, as reported by Faidon. the Commons/GLAM communities are waiting for a response from us to resume turn requested our feedback at: https://commons.wikimedia.org/wiki/Commons:Village_pump#Images_so_big_they_… currently handle, and we invite ideas on how the strain on the servers for the large images involved can be reduced. the VipsScaler for TIFF pictures and Greg has already worked on this and proposed a patch. But this patch has been waiting a review since 7 months: wrote: https://commons.wikimedia.org/wiki/Commons:Village_pump#Images_so_big_they_… likely cause of the problem with us. related to Upload Wizard — and this seems like a good one for us to take on. is likely to take most of our attention for the next few weeks. tickets about this issue, so we can hit the ground running in early may. understanding. emmanuel.engelhart(a)wikimedia.ch> wrote: processes, so have >>>>>> more info. >>>>>>> Indeed, I do: this had nothing to do with TMH. The trigger was Commons >>>>> User:Fæ uploading hundreds of 100-200MB multipage TIFFs via GWToolset >>>>> over the course of 4-5 hours (multiple files per minute), and then >>>>a >>>>> thumbnail for all of those new files in parallel in realtime, and thus >>>>inadvertently user then https://commons.wikimedia.org/wiki/User_talk:Fæ the https://commons.wikimedia.org/wiki/Commons:Village_pump#Images_so_big_they_… do completely serious issue, in particular for Wikipedians in Residence. To me it looks like that using the Vipsscaler would help to fix the worse ones. it public with the hope it might help. institutions to upload high quality pictures to increase digital sustainability. But, in the worse case (big TIFF files), Commons is not able to deal with them and fails to compute the thumbnails. collection of historical plans of the Zurich main station: https://commons.wikimedia.org/wiki/Category:Historical_plans_of_Zurich_Main… >>>> It seems that this problem might be fixed by using the VipsScaler for TIFF pictures and Greg has already worked on this and proposed a patch. But this patch has been waiting a review since 7 months: >>>> https://bugzilla.wikimedia.org/show_bug.cgi?id=52045 >>>> IMO it would be great if you could do something to increase the priority and the urgency of this ticket. The movement invests pretty much resources to build successful collaboration with GLAMs and many of them get braked by this "silly" bug. >>>> Hope you can help us. >>>> =========== >>>> -- >>>> Volunteer >>>> Technology, GLAM, Trainings >>>> Zurich >>>> +41 797 670 398 >>>> _______________________________________________ >>>> Multimedia mailing list >>>> Multimedia(a)lists.wikimedia.org >>>> https://lists.wikimedia.org/mailman/listinfo/multimedia >>> >>> >>> _______________________________ >>> >>> Fabrice Florin >>> Product Manager >>> Wikimedia Foundation >>> >>> http://en.wikipedia.org/wiki/User:Fabrice_Florin_(WMF) >>> >>> >>> >> >>> _______________________________________________ >>> Ops mailing list >>> Ops(a)lists.wikimedia.org >>> https://lists.wikimedia.org/mailman/listinfo/ops >> >> > > _______________________________ > > Fabrice Florin > Product Manager > Wikimedia Foundation > > http://en.wikipedia.org/wiki/User:Fabrice_Florin_(WMF) > > > > > _______________________________________________ > Multimedia mailing list > Multimedia(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/multimedia >

Thanks for the hotfix, Aaron, and the reply to my questions, Faidon. The multimedia team has allocated time this sprint to find a short-term and long-term plan for the general Image Scaler situation, and each subsequent sprint we'll have time allocated as well to act on that plan and implement it. I'd like to have an engineering meeting with both of you and our team (invite sent), to help us fully understand the various parts involved. At the moment I think we all (multimedia folks) have a lot of catching up to do in terms of knowledge of that code, I don't think any of us would have been capable of writing the hotfix Aaron did. Our team needs to acquire that knowledge ASAP. Now, let's not wait for the meeting to start discussing the issue and proposed solutions. As I've stated earlier in this thread, I would like to revisit the fact that the thumbnails are being generated in real time, because it's the bigger problem and Image Scalers will still be subject to going down if we don't fix that. I suspect that a solution to that broader problem would solve the GWToolset issue as well. Aaron worked on wrapping thumb calls in PoolCounter on the MediaWiki side Am I correct to assume that the PoolCounter wrapping is used to dedupe requests from different image scaler servers to generate the same thumbnail? I've thought about the problem since the incident and I came up with the following rough idea, that I'd like everyone to criticize/validate against their knowledge of the existing setup. Note that it's based on a surface understanding of things, I still don't know the existing code very well, which is why your knowledge applied against my idea will be very useful. Instead of each image scaler server generating a thumbnail immediately when a new size is requested, the following would happen in the script handling the thumbnail generation request: - the client's connection would be kept open, potentially for a while. All the way from the end-user to the thumb script - a data store (what kind is TBD, whatever is most appropriate, redis might be a good candidate) would be connected to and a virtual resource request would be added to a queue. Basically the script would be queueing an "I need one unit of thumbnail generation" request object - in a loop, the same data store would be read every X milliseconds to check the request object's position in the queue - if the request object's position is below a certain configurable limit, the actual job starts, scaling the image and storing the thumb - when the script is done generating the thumbnail, it removes its request object from the queue, essentially liberating a virtual unit of thumbnail generation ork for another thumb request to use. And of course it returns the generated thumb as it currently does. - if the script dies for whatever reason, an expiry setting on the queued request object would kick in at the data store level, liberating the virtual unit automatically - if the client requesting the thumb gives up and closes the connection because there's a high load and they don't see the thumb appear fast enough to their taste, the script would keep running, wait for its turn in the queue and ultimately render the thumb anyway. This is crucial, because at times of high load when things start taking time, we definitely don't want users refreshing the page to result in aborting and restarting the exact same thumb generation work. Every thumb generation request should complete, even if much later and with no end-user with an open connection to see the result anymore. Pros: - The limit on virtual units of work means that the image scaler server load should never go over a certain point. No more going down depending on the mix or quantity of thumb requests coming in. The servers would just be handling as much work as they actually can and no more. - This would be agnostic of how long a given thumbnail generation takes. Which means that someone uploading lots of large files that are very time-consuming to generate thumbs for, such as last weekend's incident, wouldn't take down image scalers. They'd just slow down thumb generation across the board. - The queue size could be configurable. The best strategy is to start low (eg. as many units as there are servers) - The queue could be smarter than a plain queue and have weight and client-based priority strategies. For example we could make it so that people uploading a lot of large images don't hog the queue to themselves. I don't see that as a requirement to solve the reliability issues, but it would be nice to have, and such a prioritization would be very relevant to the GWToolset user behavior. It would basically that someone uploading a lot of large images would only have their thumbs take longer to generate than usual, not everyone else's as well. - We could write "attack bots" that would attempt DoSing the image scalers in various ways and let us verify that this new system behaves well under heavy load. We know the weaknesses, we should be testing them and making sure that they're solved, instead of waiting for someone with bad intentions or some accident coming from unusual usage pattern to do that for us. Cons: - Under high load, HTTP connections could potentially be held for a while. Maybe that's not such a big deal, I guess Ops has the answer. I think it's fair to set a time limit on that too, and still have the thumbnail generation happen even if we closed the connection to the client. From the end-user's perspective, the experience would be that they wait a while for a thumb to appear, the connection dies, but chances are that if they refresh the page a few seconds or minutes later, the thumb will be there. - During high load/DoS attack/whatever people viewing thumbs at new sizes will experience long image load times. Considering that most of the time, the thumb will end up appearing before a timeout occurs, I think that's preferable to the status quo, which is that image scalers would go down entirely in those situations. On Tue, Apr 22, 2014 at 1:31 PM, Faidon Liambotis <faidon(a)wikimedia.org>wrote;wrote:

This is similar to what I proposed to Ori. For multi-page file (pdf,djvu,tiff) we'd prerender base thumbnails and use them for downscaling in thumb.php on demand. The base thumbnails would only be so large (e.g. not 10000px width) since there isn't much use case for massive thumbnails vs just viewing the original. This would also apply to single page TIFFs, where one reference thumbnail of reasonable size would be used for downscaling on demand. The reference files could be created on upload before the file could even appear at places like Special:NewImages. Resizing the reference files would actually be reasonable to do in thumb.php. File purges could exempt the reference thumbnails themselves (or if it didn't then their generation would be pool countered like TMH does at least). The reference thumbnails should also be in Swift even if we move to CDN "only" thumbnail storage. Disk space is cheap enough for this. On Thu, Apr 24, 2014 at 8:57 AM, Gabriel Wicke <gwicke(a)wikimedia.org> wrote: -- -Aaron S

I changed the failure limiting to detecting fatals in https://gerrit.wikimedia.org/r/#/c/127642/ and also tweaked the "large source file download" pool counter config timeouts in https://gerrit.wikimedia.org/r/#/c/127654/. Hopefully that should help for now. On Mon, Apr 21, 2014 at 3:05 AM, Faidon Liambotis <faidon(a)wikimedia.org>wrote;wrote: -- -Aaron S

On Sun, May 11, 2014 at 5:04 AM, Faidon Liambotis <faidon(a)wikimedia.org>wrote;wrote: I think the short-term outcome was to throttle GWToolset until there is a better fix. There is a patch pending to do that: https://gerrit.wikimedia.org/r/#/c/132111/ https://gerrit.wikimedia.org/r/#/c/132112/ I described the thinking behind the limits in this mail and the followups: http://thread.gmane.org/gmane.org.wikimedia.glamtools/24/focus=104 tl;dr it tries to limit the GWToolset-uploaded thumbnails appearing in Special:* at one time to 10% (5 with default settings), based on the total upload rate in the slowest hour of an average day. That's about one image per two minutes. This is not live yet; Fae has apparently been running the upload with three threads (as a side note, GWToolset should probably log this), which means about one image per 20 sec (more precisely, three upload jobs are generated per minute; how the job queue picks them is not controlled). (I haven't been paying the issue much attention because I assumed that GWToolset is inoperational anyway until the next deploy due to bug 64931. Apparently I misunderstood how the job queue handles jobs that throw exceptions.)

On Tue, May 13, 2014 at 2:45 PM, Gergo Tisza <gtisza(a)wikimedia.org> wrote: Maybe we could create scaling priority groups? Tag GWToolset-uploaded images as belonging to the "expensive" group, then use PoolCounter to ensure that no more than X expensive thumbnails are scaled at the same time. That would throttle thumbnail rendering directly, instead of throttling the upload speed and making guesses about how that translates a throttle on rendering.

On Tue, May 13, 2014 at 3:29 PM, Aaron Schulz <aschulz(a)wikimedia.org> wrote: For files which belong to a priority group, instead of using the file name as a key, couldn't we just hash the filename into one of the available slots for that group? That would still ensure only one process per file, but it would also limit the number of files processed from the whole group. The pool would be underused due to hash conflicts when the number of files waiting is not significantly larger than the pool itself, but that doesn't seem like a huge problem for a small pool.

On Tue, May 13, 2014 at 02:45:21PM -0700, Gergo Tisza wrote: Sort of, yes. CPU was spiking but also MaxClient limits were being hit, as requests were piling up due to the large duration of those large scaling requests, both due to the time it takes to actually run it, as well as due to the time it takes to actually transfer the files. It's unlikely we'd survive this if we increased MaxClients, though. The former. The network spike graphs correlated exactly with equivalent imagescaler network spike graphs. Note that this has a secondary effect: when the network gets saturated, imagescaler original transfers become slower and hence scaling requests pile up (see above). Not exactly, no. I found 34 XML files in Swift under the container wikipedia-commons-gwtoolset-metadata starting with "Fæ/". A "grep -hr '<filename>' | sort -u | wc -l" showed 7396 *files* (all containing "NYPL" in the name) that would be eventually be uploaded (AIUI), not jobs in the job queue. I'm not exactly sure how GWToolset maps these into jobs in the jobqueue, but I remember reading something about a "master" job that uploads multiple files when it runs. That sounds acceptable to me, preferrably in combination with the reference thumbnail idea. Faidon