On Fri, Nov 15, 2013 at 12:20 AM, Yury Katkov <katkov.juriy(a)gmail.com> wrote:
Just don't tell me that the 6th most popular
website on Earth don't do
any load testing! Maybe I don't understand the process and you test
the software in a completely different way?
We load-test by having a staggered deployment process for new code.
New code is snapshotted as a branch in Git and then pushed out to a
low-traffic subset of Wikimedia wikis at first. Over the course of the
following week, the set of wikis on which the code is deployed is
gradually expanded to include more high-traffic wikis, until it is
eventually deployed across the entire cluster. We monitor load on the
cluster through the process, using Ganglia
(<http://ganglia.wikimedia.org/latest/>) and Graphite
(<http://gdash.wikimedia.org/>). Profiling data emitted by MediaWiki
itself is tagged with the name of the deployment branch, which helps
us correlate fluctuations in load with specific code changes:
<http://noc.wikimedia.org/cgi-bin/report.py>.
The deployment cycle is explained in greater detail at
<https://wikitech.wikimedia.org/wiki/Deployments> and
<https://wikitech.wikimedia.org/wiki/Deployments/One_week>. The
description I provided above is not exhaustive; it omits, for example,
the time that new code changes spend on the WMF Beta Cluster, which is
also instrumented for performance monitoring.
Hope this helps,
Ori