On Fri, Nov 15, 2013 at 12:20 AM, Yury Katkov katkov.juriy@gmail.com wrote:

Just don't tell me that the 6th most popular website on Earth don't do any load testing! Maybe I don't understand the process and you test the software in a completely different way?

We load-test by having a staggered deployment process for new code. New code is snapshotted as a branch in Git and then pushed out to a low-traffic subset of Wikimedia wikis at first. Over the course of the following week, the set of wikis on which the code is deployed is gradually expanded to include more high-traffic wikis, until it is eventually deployed across the entire cluster. We monitor load on the cluster through the process, using Ganglia (http://ganglia.wikimedia.org/latest/) and Graphite (http://gdash.wikimedia.org/). Profiling data emitted by MediaWiki itself is tagged with the name of the deployment branch, which helps us correlate fluctuations in load with specific code changes: http://noc.wikimedia.org/cgi-bin/report.py.

The deployment cycle is explained in greater detail at https://wikitech.wikimedia.org/wiki/Deployments and https://wikitech.wikimedia.org/wiki/Deployments/One_week. The description I provided above is not exhaustive; it omits, for example, the time that new code changes spend on the WMF Beta Cluster, which is also instrumented for performance monitoring.

Hope this helps, Ori