[QA] run browser tests against production Wikipedia?

[Greg Grossmeier]

<quote name="Nikolas Everett" date="2013-09-19" time="16:57:35 -0400">
> On Thu, Sep 19, 2013 at 4:55 PM, Greg Grossmeier <greg at wikimedia.org> wrote:
> > <quote name="Rachel Thomas" date="2013-09-19" time="16:26:39 -0400">
> >> In that case, I can see why Zeljko removed them.  I guess the next step
> >> would be to think of what deliberate tests we should run against production.
> >
> > +1
> >
> > What are good canaries for larger issues on production?
> 
> Another way to think about this is what are things that we really
> really really want to notice fast if they break.  For instance, the
> main page of any wiki stops working, we should know.

My understanding/assumption is that these aren't run very frequently;
something like 3 times/day.

What's the phrase? Any sufficiently advanced testing is
indistinguishable from monitoring? The other way around? whatever :)

I think I'd focus on:

What are the features that are important but aren't easily monitored
through ganglia/whatever?

What set of tests can we use to ensure there aren't oddities on real
production sites that aren't there on the fake production sites like
test2?


Greg

-- 
| Greg Grossmeier            GPG: B2FA 27B1 F7EB D327 6B8E |
| identi.ca: @greg                A18D 1138 8E47 FAC8 1C7D |