On Tue, Mar 26, 2013 at 3:58 PM, Asher Feldman
<afeldman(a)wikimedia.org>wrote;wrote:
There are all good points, and we certainly do
need better tooling for
individual developers.
There are a lot of things a developer can do on just a laptop in terms of
profiling code, that if done consistently, could go a long way, even
without it looking anything like production. Things like understanding if
algorithms or queries are O(n) or O(2^n), etc. and thinking about the
potential size of the relevant production data set might be more useful at
that stage than raw numbers. When it comes to gathering numbers in such an
environment, it would be helpful if either the mediawiki profiler could
gain an easy visualization interface appropriate for such environments, or
if we standardized around something like xdebug.
The beta cluster has some potential as a performance test bed if only it
could gain a guarantee that the compute nodes it runs on aren't
oversubscribed or that the beta virts were otherwise consistently
resourced. By running a set of performance benchmarks against beta and
production, we may be able to gain insight on how new features are likely
to perform.
This is possible in newer versions of OpenStack, using scheduler hinting.
That said, the instances would still be sharing a host with each other,
which can cause some inconsistencies. We'd likely not want to use beta
itself, but something that has limited access for performance testing
purposes only, as we wouldn't want other unrelated testing load to skew
results. Additionally, we'd want to make sure to avoid things like
/data/project or /home (both of which beta is using), even once we've moved
to a more stable shared storage solution, as it could very heavily skew
results based on load from other projects.
We need to upgrade to the Folsom release or greater for a few other
features anyway, and enabling scheduler hinting is pretty simple. I'd say
let's consider adding something like this to the Labs infrastructure
roadmap once the upgrade happens and we've tested out the hinting feature.
I am concerned in this discussion with insufficient testbed load
generation and avoidance of confounding variables in the performance
analysis...
--
-george william herbert
george.herbert(a)gmail.com