It's less about it being not appropriate for perf tests and more so
that you can't directly compare bare metal to a virtual instance. Load
testing virts against virts is fine as long as your factor the drift
in shared resourcing.

Agree that it's not apples to apples, but can't you still get average memory and/or CPU usage?   I only suggested it since I thought it would be a cheaper way to run a cursory assessment, which is all I assumed we needed at this point.

This is also why I mentioned gradual rollout and feature flags in my earlier reply. Ideally we can defer establishing performance SLAs until after we're ready to promote this from "experiment" to a first-class stack citizen.

Anyway, on with the experiment! Definitely excited for us to learn more about this as a team (and organization).

On Tue, Mar 10, 2015 at 2:02 PM, Tomasz Finc <tfinc@wikimedia.org> wrote:
On Tue, Mar 10, 2015 at 10:49 AM, Max Semenik <msemenik@wikimedia.org> wrote:
> As a shared virtualized environment, Labs is not appropriate for performance
> measurements. Also, its SLA is too lax for production app use.

Moving back to mobile-l@

It's less about it being not appropriate for perf tests and more so
that you can't directly compare bare metal to a virtual instance. Load
testing virts against virts is fine as long as your factor the drift
in shared resourcing.

As for LABS production use I'm curious about would it would take to
make this work in the future. LABS is a low cost env to spin up
hardware that is easily discarded after. To me this is critical in
supporting our engineers teams experiment with low cost.

I know that isn't something that we do currently but either we need to
get production into a state where its simpler to spin up new
production instances/products/extensions/etc or we allow LABS to take
some small production traffic for experiments.

CC'ing Yuvi to get his thoughts.

--tomasz



--
EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle
IRC: bgerstle