[QA] [WikimediaMobile] big improvements to the MobileFrontend browser test repo

[Chris McMahon]

On Fri, Dec 5, 2014 at 2:23 AM, Joaquin Oltra Hernandez <
jhernandez at wikimedia.org> wrote:
>
> Here you can see vagrant roles, final output of the tests and env
> variables https://gist.github.com/joakin/eae3b086dd5a0c498669
>
> I'm running the tests with phantomjs by default for convenience and speed,
> but I've tried Firefox and I've gotten similar results.
>
> Anything I've missed that I should do? I'd really love to get them green
> to rely on them more often and avoid regressions.
>

Sure, I see a few issues right off, even without looking at how these
failed.

* Uploads has been disabled in the MobileFrontend for a long time. That
feature may or may not be re-enabled in the future. That test is simply not
valid, but I have kept it in the repo at the request of the Mobile team.
You can see in the tags in special_upload.feature and
uploads_copyvio.feature that those tests have no target environments and do
not run in the build.

For the rest, these tests have been designed to work primarily in beta
labs.  If you set your env vars to

export MEDIAWIKI_URL=http://en.m.wikipedia.beta.wmflabs.org/wiki/

export MEDIAWIKI_API_URL=http://en.wikipedia.beta.wmflabs.org/w/api.php

export MEDIAWIKI_USER=Selenium_user

export MEDIAWIKI_PASSWORD=(password)

export BROWSER=firefox
or equivalent, I bet the suite would pass (except for the uploads tests).

* Do you have the API URL set correctly? That would be a source of failure.
* Do you have VisualEditor in place? If not, the VE tests will fail.
* The language tests rely on having the page available in another language
wiki, and the db records to show it.
* Do you have Nearby enabled?  Search? etc?

Each test failure will display a detailed message describing what condition
the test expected and what condition violated that expectation.  I think if
you look at the detailed failure message it will tell you exactly why the
test failed in your environment.

-Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/qa/attachments/20141205/3f3a0848/attachment.html>