Some of you may have noticed a bot [1] providing reviews for the
Mobilefrontend and Gather extensions.
This is a grass routes experiment [2] to see if we can reduce
regressions by running browser tests against every single commit. It's
very crude, and we're going to have to maintain it but we see this as
a crude stop gap solution until we get gerrit-bot taking care of this
for us.
Obviously we want to do this for all extensions but we wanted to get
something good enough that is not scaleable to start exploring this.
So far it has caught various bugs for us and our browser test builds
are starting to finally becoming consistently green, a few beta labs
flakes aside [3].
Running tests on beta labs is still useful but now we can use it to
identify tests caused by other extensions. We were finding too often
our tests were failing due to us neglecting them.
In case others are interested in how this is working and want to set
one up themselves I've documented this here:
https://www.mediawiki.org/wiki/Reading/Setting_up_a_browser_test_bot
Please let me now if you have any questions and feel free to edit and
improve this page. If you want to jump into the code that's doing this
and know Python check out:
https://github.com/jdlrobson/Barry-the-Browser-Test-Bot
(Patches welcomed and apologies in advance for the code)
[1]
https://gerrit.wikimedia.org/r/#/q/reviewer:jdlrobson%252Bbarry%2540gmail.c…
[2]
https://phabricator.wikimedia.org/T100293
[3]
https://integration.wikimedia.org/ci/view/Mobile/job/browsertests-MobileFro…