[QA] Two big improvements for flaky tests

[James Forrester]

On 25 July 2014 09:29, Chris McMahon <cmcmahon at wikimedia.org> wrote:

>
> This month we've made two big improvements that should greatly reduce the
> number of flaky test failures in all of the builds.
>
> For one thing, the MobileFrontend test that used to walk through a dozen
> steps to protect a page in the UI and then logout, is now using the API to
> protect the test page, and not logging out prevents flaky failures when
> another test that expects to be logged-in unexpectedly gets logged out.  We
> have seen this before in VisualEditor tests in particular.
>
> Another thing is that we should no longer see the beta labs db become
> read-only.  Some time ago we had requests from the Language team to be able
> to test in situations where there is replication lag between master and
> slave dbs.  At the same time, the Flow team wanted to be able to test
> deploying in a situation with master and slave dbs. So we created a slave
> db for beta labs.
>
> What we only discovered recently is that there is a process that monitors
> the replication lag time between master and slave dbs, and when that time
> goes over 5 seconds, the db is set to read-only. This is fine in
> production, but it was causing a lot of problems in beta labs.  We have now
> set the value for replication lag to 5 minutes instead of 5 seconds, and
> this has stopped a number of kinds of faillures:
>
> * the explicit "database is read-only" failure from the VisualEditor tests
> * the generic "Save failed" failure message in MobileFrontend tests
> * the unexplained 90-second timeouts in Flow tests
>
> Sorry for the inconvenience, but things should be much improved from now
> on.
>

​Ha. Wow. Well, yes, this sounds very promising for the tests to have fewer
random gremlins! Thanks.

J.
-- 
James D. Forrester
Product Manager, Editing
Wikimedia Foundation, Inc.

jforrester at wikimedia.org | @jdforrester
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/qa/attachments/20140725/0ccefe6a/attachment.html>