[QA] Holding our code to better standards.

[Jon Robson]

On Thu, Sep 3, 2015 at 1:59 PM, Greg Grossmeier <greg at wikimedia.org> wrote:
> (I've put the TO: field as the QA list only, and put everyone else on
> BCC now. If you're curious in this topic, please join the QA mailing
> list and follow along. It's not a very high traffic list.)
>
> <quote name="Jon Robson" date="2015-09-03" time="11:45:47 -0700">
>> Dear Greg, and anyone else that is involved in deployment
>
> Hi there :)
>
>> <successes over the past month>
>
> Awesome :)
>
>> The future!:
>> Given this success:
>> 1) I would like to see us run @integration tests on core, but I
>> understand given the number of bugs this might not be feasible so far.
>
> https://integration.wikimedia.org/ci/view/BrowserTests/view/Core/
>
> Those are pretty stable, but limited (rightfully) in number. Looks like
> the last run of the first job took 8 minutes. Not too bad.
Yup. Have set up https://phabricator.wikimedia.org/T111467 to make this happen.


>> 2) We should run @integration tests prior to deployments to the
>> cluster via the train and communicate out when we have failures (and
>> make a decision to push broken code)
>
> The way I hear this is: Run @integration tests on merge to wmfXX
> branches. Is that an accurate rephrasing?

I'm not too clear with how we currently do deployments but as I
understand it we manually cut a branch for wmf. I'd like part of this
process to involve triggering all the @integration tests and
generating a url that we can share with each deployment.

We may want to run it during SWATs too but right now I'm just keen to
get this working on big deployments (I'd hope we're not making big
SWATs that are capable of devastating our projects :))

>
> If not, then it mean what we're planning on doing with respect to the
> "Staging" cluster work we started, paused (due to time constraints), and
> will restart in Q3. tl;dr: The staging cluster will be a cluster that
> runs nightly full blown tests against a git tag. In the morning we'll be
> able to make a go/no-go decision on deploying that tag.

That sounds great and what I'm keen to see happen.

>
> That "nightly" part can, of course, be modified to whatever frequency we
> can support (which can probably be pretty fast).

I think at minimum it just needs to run once (even if triggered
manually) before we push out to wikis.

>
>> 3) I'd like to see other extensions adopt browser test voting on their
>> extensions. Please feel free to reach out to me if you need help with
>> that. The more coverage across our extensions we have, the better.
>
> Thanks for the offer of help, Jon! That's awesome. I love the idea of
> teams/groups helping other teams/groups! You, personally, have been
> great this so far and I thank you for that.

I'm here if needed. The more extensions relying on browser tests the
less fire fighting I have to do.

>
> Greg
>
> --
> | Greg Grossmeier            GPG: B2FA 27B1 F7EB D327 6B8E |
> | identi.ca: @greg                A18D 1138 8E47 FAC8 1C7D |
>
> _______________________________________________
> QA mailing list
> QA at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/qa