On Fri, Nov 15, 2013 at 12:20 AM, Yury Katkov katkov.juriy@gmail.com wrote:

Just don't tell me that the 6th most popular website on Earth don't do any load testing! Maybe I don't understand the process and you test the software in a completely different way?

We load-test by having a staggered deployment process for new code. New code is snapshotted as a branch in Git and then pushed out to a low-traffic subset of Wikimedia wikis at first. Over the course of the following week, the set of wikis on which the code is deployed is gradually expanded to include more high-traffic wikis, until it is eventually deployed across the entire cluster. We monitor load on the cluster through the process, using Ganglia (http://ganglia.wikimedia.org/latest/) and Graphite (http://gdash.wikimedia.org/). Profiling data emitted by MediaWiki itself is tagged with the name of the deployment branch, which helps us correlate fluctuations in load with specific code changes: http://noc.wikimedia.org/cgi-bin/report.py.

The deployment cycle is explained in greater detail at https://wikitech.wikimedia.org/wiki/Deployments and https://wikitech.wikimedia.org/wiki/Deployments/One_week. The description I provided above is not exhaustive; it omits, for example, the time that new code changes spend on the WMF Beta Cluster, which is also instrumented for performance monitoring.

Hope this helps, Ori

On Nov 15, 2013, at 12:33 PM, Tyler Romeo tylerromeo@gmail.com wrote:

Isn't it always best just to go with the flow? ;)

No.

Build performance testing and QA testing into the flow on any project.

(I know you were joking, but the industry as a whole does not get this one, so I berate people a lot...).

-george william herbert george.herbert@gmail.com

Sent from Kangphone