On Fri, Dec 5, 2014 at 2:23 AM, Joaquin Oltra Hernandez <jhernandez@wikimedia.org> wrote:

Here you can see vagrant roles, final output of the tests and env variables https://gist.github.com/joakin/eae3b086dd5a0c498669

I'm running the tests with phantomjs by default for convenience and speed, but I've tried Firefox and I've gotten similar results.

Anything I've missed that I should do? I'd really love to get them green to rely on them more often and avoid regressions.

Sure, I see a few issues right off, even without looking at how these failed.

* Uploads has been disabled in the MobileFrontend for a long time. That feature may or may not be re-enabled in the future. That test is simply not valid, but I have kept it in the repo at the request of the Mobile team. You can see in the tags in special_upload.feature and uploads_copyvio.feature that those tests have no target environments and do not run in the build.

For the rest, these tests have been designed to work primarily in beta labs. If you set your env vars to

export MEDIAWIKI_URL=http://en.m.wikipedia.beta.wmflabs.org/wiki/

export MEDIAWIKI_API_URL=http://en.wikipedia.beta.wmflabs.org/w/api.php

export MEDIAWIKI_USER=Selenium_user

export MEDIAWIKI_PASSWORD=(password)

export BROWSER=firefox

or equivalent, I bet the suite would pass (except for the uploads tests).

* Do you have the API URL set correctly? That would be a source of failure.

* Do you have VisualEditor in place? If not, the VE tests will fail.

* The language tests rely on having the page available in another language wiki, and the db records to show it.

* Do you have Nearby enabled? Search? etc?

Each test failure will display a detailed message describing what condition the test expected and what condition violated that expectation. I think if you look at the detailed failure message it will tell you exactly why the test failed in your environment.

-Chris