On Tue, Sep 1, 2015 at 4:54 PM, Brandon Black bblack@wikimedia.org wrote:
I really do like the idea of moving towards smarter ratelimiting of APIs by default, though (and have brought this up in several contexts recently, but I'm not really aware of whatever past work we've done in that direction). From that relatively-ignorant perspective, I tend to envision an architecture where the front edge ratelimits API requests (or even possibly, all requests, but we'd probably have to exclude a lot of common spiders...) via a simple token-bucket-filter if they're anonymous, but lets them run free if they superficially appear to have a legitimate cookie or API access token. Then it's up to the app layer to enforce limits for the seemingly-identifiable traffic and be configurable to raise them for legitimate remote clients we've had contact with, and to reject legitimate-looking tokens/logins that the edge choses not to ratelimit which aren't actually legitimate.
Rate limiting / UA policy enforcement has to be done in Varnish, since API responses can be cached there and so the requests don't necessarily reach higher layers (and we wouldn't want to vary on user agent).