On Tue, Sep 15, 2015 at 6:56 PM, Marko Obrovac mobrovac@wikimedia.org wrote:
On 15 September 2015 at 19:37, Dan Andreescu dandreescu@wikimedia.org wrote:
I worry a little bit about the performance without having a batch api,
but we can certainly try it out and see what happens. Basically we will be requesting the page view information for every NS_MAIN article in every wiki once a week. A quick sum against our search cluster suggests this is ~96 million api requests.
96m equals approx 160 req/s which is more than sustainable for RESTBase.
True, if we distributed the load over the whole week, but I think Erik needs the results to be available weekly, as in, probably within a day or so of issuing the request. Of course, if we were to serve this kind of request from the API, we would make a better batch-query endpoint for his use case. But I think it might be hard to make that useful generally. I think for now, let's just collect these one-off pageview querying use cases and slowly build them into the API when we can generalize two or more of them into one endpoint.