Well the Varnish request coalescing can't work for requests with login Cookie headers. I've seen code in our VCLs to disable it in those cases since the origin response cannot be reused for authenticated requests typically.
On Tue, Apr 22, 2014 at 3:47 AM, Faidon Liambotis faidon@wikimedia.orgwrote:
On Mon, Apr 21, 2014 at 12:22:36PM +0200, Gilles Dubuc wrote:
Can you clarify something: for a given set of heavyweight thumbnails that need to be rendered, assuming the the uploads have ceased, would multiple visits of Special:NewFiles in a short timeframe multiply the saturation
by
the amount of HTTP requests to the same thumbnail URLs? I.e. if you
request
the URL of a thumbnail which is currently being generated because someone else requested it, does it make the issue worse?
Varnish should do some coalescing of requests with the exact same URL and Aaron worked on wrapping thumb calls in PoolCounter on the MediaWiki side lately as well. So, in theory, no, this shouldn't happen.
Note that Special:NewFiles will show thumbnails from multiple files, though, so your browser would request multiple *different* thumbnails (different URLs) in parallel.
Second question is, how come piling on jobs doesn't just make the jobs
that
came last complete much later? The same kind of DoS situation could
happen
with someone bombarding us with HEAD requests on previously unrequested thumbnail sizes for small images, so I think that the issue isn't
specific
to large jobs. It's more a matter of properly queueing things up so that the imagescalers don't overload, regardless of the mix of job weight.
Yes, there are multiple vectors here that can DoS us, including the one you mention :) Note that these aren't "jobs" in the jobqueue sense; thumb generation happens in realtime, for every request that comes for a size that isn't already stored (cached) in Swift.
Faidon
Ops mailing list Ops@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/ops