On Tue, Mar 10, 2015 at 10:49 AM, Max Semenik msemenik@wikimedia.org wrote:
As a shared virtualized environment, Labs is not appropriate for performance measurements. Also, its SLA is too lax for production app use.
Moving back to mobile-l@
It's less about it being not appropriate for perf tests and more so that you can't directly compare bare metal to a virtual instance. Load testing virts against virts is fine as long as your factor the drift in shared resourcing.
As for LABS production use I'm curious about would it would take to make this work in the future. LABS is a low cost env to spin up hardware that is easily discarded after. To me this is critical in supporting our engineers teams experiment with low cost.
I know that isn't something that we do currently but either we need to get production into a state where its simpler to spin up new production instances/products/extensions/etc or we allow LABS to take some small production traffic for experiments.
CC'ing Yuvi to get his thoughts.
--tomasz
It's less about it being not appropriate for perf tests and more so that you can't directly compare bare metal to a virtual instance. Load testing virts against virts is fine as long as your factor the drift in shared resourcing.
Agree that it's not apples to apples, but can't you still get average memory and/or CPU usage? I only suggested it since I thought it would be a cheaper way to run a cursory assessment, which is all I assumed we needed at this point.
This is also why I mentioned gradual rollout and feature flags in my earlier reply. Ideally we can defer establishing performance SLAs until after we're ready to promote this from "experiment" to a first-class stack citizen.
Anyway, on with the experiment! Definitely excited for us to learn more about this as a team (and organization).
On Tue, Mar 10, 2015 at 2:02 PM, Tomasz Finc tfinc@wikimedia.org wrote:
On Tue, Mar 10, 2015 at 10:49 AM, Max Semenik msemenik@wikimedia.org wrote:
As a shared virtualized environment, Labs is not appropriate for
performance
measurements. Also, its SLA is too lax for production app use.
Moving back to mobile-l@
It's less about it being not appropriate for perf tests and more so that you can't directly compare bare metal to a virtual instance. Load testing virts against virts is fine as long as your factor the drift in shared resourcing.
As for LABS production use I'm curious about would it would take to make this work in the future. LABS is a low cost env to spin up hardware that is easily discarded after. To me this is critical in supporting our engineers teams experiment with low cost.
I know that isn't something that we do currently but either we need to get production into a state where its simpler to spin up new production instances/products/extensions/etc or we allow LABS to take some small production traffic for experiments.
CC'ing Yuvi to get his thoughts.
--tomasz
On Tue, Mar 10, 2015 at 11:02 AM, Tomasz Finc tfinc@wikimedia.org wrote:
It's less about it being not appropriate for perf tests and more so that you can't directly compare bare metal to a virtual instance. Load testing virts against virts is fine as long as your factor the drift in shared resourcing.
As for LABS production use I'm curious about would it would take to make this work in the future. LABS is a low cost env to spin up hardware that is easily discarded after. To me this is critical in supporting our engineers teams experiment with low cost.
I know that isn't something that we do currently but either we need to get production into a state where its simpler to spin up new production instances/products/extensions/etc or we allow LABS to take some small production traffic for experiments.
The ops team is currently working on virtualization in production. How easy it will be in practice is another question, because currently there's too many hurdles in bringing up a new service, and allocating the hardware (which will be alleviated by virtualization) is just one part of problem that also includes monitoring, getting ops approvals for many things, requesting access to the new servers for developers, etc.