On Tue, Mar 26, 2013 at 8:15 PM, Ryan Lane rlane32@gmail.com wrote:
On Tue, Mar 26, 2013 at 3:58 PM, Asher Feldman afeldman@wikimedia.orgwrote:
There are all good points, and we certainly do need better tooling for individual developers.
There are a lot of things a developer can do on just a laptop in terms of profiling code, that if done consistently, could go a long way, even without it looking anything like production. Things like understanding if algorithms or queries are O(n) or O(2^n), etc. and thinking about the potential size of the relevant production data set might be more useful at that stage than raw numbers. When it comes to gathering numbers in such an environment, it would be helpful if either the mediawiki profiler could gain an easy visualization interface appropriate for such environments, or if we standardized around something like xdebug.
The beta cluster has some potential as a performance test bed if only it could gain a guarantee that the compute nodes it runs on aren't oversubscribed or that the beta virts were otherwise consistently resourced. By running a set of performance benchmarks against beta and production, we may be able to gain insight on how new features are likely to perform.
This is possible in newer versions of OpenStack, using scheduler hinting.
That said, the instances would still be sharing a host with each other, which can cause some inconsistencies. We'd likely not want to use beta itself, but something that has limited access for performance testing purposes only, as we wouldn't want other unrelated testing load to skew results. Additionally, we'd want to make sure to avoid things like /data/project or /home (both of which beta is using), even once we've moved to a more stable shared storage solution, as it could very heavily skew results based on load from other projects.
We need to upgrade to the Folsom release or greater for a few other features anyway, and enabling scheduler hinting is pretty simple. I'd say let's consider adding something like this to the Labs infrastructure roadmap once the upgrade happens and we've tested out the hinting feature.
I am concerned in this discussion with insufficient testbed load generation and avoidance of confounding variables in the performance analysis...