API is fairly complex to meassure and performance target. If a bot requests 5000 pages in one call, together with all links & categories, it might take a very long time (seconds if not tens of seconds). Comparing that to another api request that gets an HTML section of a page, which takes a fraction of a second (especially when comming from cache) is not very useful. On Fri, Mar 22, 2013 at 1:32 AM, Peter Gehres <lists(a)pgehres.com> wrote:

There are all good points, and we certainly do need better tooling for individual developers. There are a lot of things a developer can do on just a laptop in terms of profiling code, that if done consistently, could go a long way, even without it looking anything like production. Things like understanding if algorithms or queries are O(n) or O(2^n), etc. and thinking about the potential size of the relevant production data set might be more useful at that stage than raw numbers. When it comes to gathering numbers in such an environment, it would be helpful if either the mediawiki profiler could gain an easy visualization interface appropriate for such environments, or if we standardized around something like xdebug. The beta cluster has some potential as a performance test bed if only it could gain a guarantee that the compute nodes it runs on aren't oversubscribed or that the beta virts were otherwise consistently resourced. By running a set of performance benchmarks against beta and production, we may be able to gain insight on how new features are likely to perform. Beyond due diligence while architecting and implementing a feature, I'm actually a proponent of testing in production, albeit in limited ways. Not as with test.wikipedia.org which ran on the production cluster, but by deploying a feature to 5% of enwiki users, or 10% of pages, or 20% of editors. Once something is deployed like that, we do indeed have tooling available to gather hard performance metrics of the sort I proposed, though they can always be improved upon. It became apparent that ArticleFeedbackV5 had severe scaling issues after being enabled on 10% of the articles on enwiki. For that example, I think it could have been caught in an architecture review or in local testing by the developers that issuing 17 database write statements per submission of an anonymous text box that would go at the bottom of every wikipedia article was a bad idea. But it's really great that it was incrementally deployed and we could halt its progress before the resulting issues got too serious. That rollout methodology should be considered a great success. If it can become the norm, perhaps it won't be difficult to get to the point where we can have actionable performance standards for new features, via a process that actually encourages getting features in production instead of being a complicated roadblock. On Fri, Mar 22, 2013 at 1:20 PM, Arthur Richards <arichards(a)wikimedia.org>wrote;wrote:

On Tue, Mar 26, 2013 at 8:15 PM, Ryan Lane <rlane32(a)gmail.com> wrote: I am concerned in this discussion with insufficient testbed load generation and avoidance of confounding variables in the performance analysis... -- -george william herbert george.herbert(a)gmail.com

Asher, I don't know the actual perf statistics just yet. With the API this has to be a balance - I would want more "slower" calls than tons of very fast calls - as that consumes much more bandwidth and resources (consider getting all items one item at a time - very quick, but very inefficient). On Tue, Mar 26, 2013 at 2:57 PM, Asher Feldman <afeldman(a)wikimedia.org>wrote;wrote:

Graphite refers to the cluster performance logger available at: http://graphite.wikimedia.org/ Anyone with a labs account can view it -- which as a commiter you do (it's the same as your Gerrit login.) otoh i barely ever look at the local profiling i did set up... This problem still exists with graphite; you have to look at it for it to do any good :) ~Matt Walker Wikimedia Foundation Fundraising Technology Team On Fri, Mar 22, 2013 at 2:17 PM, Brian Wolff <bawolff(a)gmail.com> wrote: