It's currently working via a script that you can find here: /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh
It would be great to have the script in a public version control system (e.g. github?), especially for people, e.g. volunteers, who can't ssh to gather-browser-tests.eqiad.wmflabs[1]
[1] all people, who're not members of https://wikitech.wikimedia.org/wiki/Nova_Resource:Mobile-smoketests
Best, Florian
-----Original-Nachricht----- Betreff: [WikimediaMobile] [Update] Browser tests per patch Datum: Thu, 18 Jun 2015 03:27:32 +0200 Von: Jon Robson jrobson@wikimedia.org An: "QA (software quality assurance) for Wikimedia projects." qa@lists.wikimedia.org, mobile-l mobile-l@lists.wikimedia.org
Background: mobile wants to gain more confidence in its browser tests by running a subset of browser tests on a case by case basis [0].
Good news: I've got a proof of concept running and Barry the browser test bot has given some legitimate helpful reviews to Gather [1].
Even better news: It's proving itself valuable already [2]. As you can see in the messages the bot has posted on [3] we have a couple of options on display option format for his reviews.
So.. hopefully this short experience has sold you all already.
This script is currently a manual job and needs a bit of tweaking before we can put it in a cron job/run it always - it needs to watch for new commits and then run a modification of the above script on a per case basis (if two versions of it run in parallel we have an issue).
Definitely something we should push for next sprint!
Long live Barry bot!
Devs... (everyone else now of what follows is likely to be useful): I got the labs instance up and running on: http://gather-browser-tests.wmflabs.org/wiki/Main_Page
Most of you in readership team should be able to ssh gather-browser-tests.eqiad.wmflabs Let me know if you have no access.
It's currently working via a script that you can find here: /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh
[0] https://phabricator.wikimedia.org/T100293 [1] https://gerrit.wikimedia.org/r/#/q/reviewer:jdlrobson%252Bbarry%2540gmail.co... [2] https://gerrit.wikimedia.org/r/#/c/218731/
_______________________________________________ Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
I agree with Florian everything that you've written should be in a public version control system.
Second, I'd ask that you document your experiences so far in getting this set up *and* how it works so that other members of the vertical can help to maintain it moving forward.
Third, great work!!1
<3
–Sam
On Thu, Jun 18, 2015 at 7:09 AM, florian.schmidt.welzow@t-online.de < florian.schmidt.welzow@t-online.de> wrote:
It's currently working via a script that you can find here: /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh
It would be great to have the script in a public version control system (e.g. github?), especially for people, e.g. volunteers, who can't ssh to gather-browser-tests.eqiad.wmflabs[1]
[1] all people, who're not members of https://wikitech.wikimedia.org/wiki/Nova_Resource:Mobile-smoketests
Best, Florian
-----Original-Nachricht----- Betreff: [WikimediaMobile] [Update] Browser tests per patch Datum: Thu, 18 Jun 2015 03:27:32 +0200 Von: Jon Robson jrobson@wikimedia.org An: "QA (software quality assurance) for Wikimedia projects." < qa@lists.wikimedia.org>, mobile-l mobile-l@lists.wikimedia.org
Background: mobile wants to gain more confidence in its browser tests by running a subset of browser tests on a case by case basis [0].
Good news: I've got a proof of concept running and Barry the browser test bot has given some legitimate helpful reviews to Gather [1].
Even better news: It's proving itself valuable already [2]. As you can see in the messages the bot has posted on [3] we have a couple of options on display option format for his reviews.
So.. hopefully this short experience has sold you all already.
This script is currently a manual job and needs a bit of tweaking before we can put it in a cron job/run it always - it needs to watch for new commits and then run a modification of the above script on a per case basis (if two versions of it run in parallel we have an issue).
Definitely something we should push for next sprint!
Long live Barry bot!
Devs... (everyone else now of what follows is likely to be useful): I got the labs instance up and running on: http://gather-browser-tests.wmflabs.org/wiki/Main_Page
Most of you in readership team should be able to ssh gather-browser-tests.eqiad.wmflabs Let me know if you have no access.
It's currently working via a script that you can find here: /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh
[0] https://phabricator.wikimedia.org/T100293 [1] https://gerrit.wikimedia.org/r/#/q/reviewer:jdlrobson%252Bbarry%2540gmail.co... [2] https://gerrit.wikimedia.org/r/#/c/218731/
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Awesome Jon! I'm so happy to finally see this developing :DD
Loving the : `Browserbot happy!`
I've noticed it can report either the name of the failing test or the full log. What do you think if we show that, and a url with the pasted log somewhere publicly to not put too much noise on the comments but still be able to see it? Something like https://phabricator.wikimedia.org/paste/...
+1 to *where is the source*. +1 to documenting how you've set it all up on wiki somewhere.
I also think we need a catchy phrase for the -1s!
Thanks for you work on this, we'll get more focused time for it soon.
On Thu, Jun 18, 2015 at 11:33 AM, Sam Smith samsmith@wikimedia.org wrote:
I agree with Florian everything that you've written should be in a public version control system.
Second, I'd ask that you document your experiences so far in getting this set up *and* how it works so that other members of the vertical can help to maintain it moving forward.
Third, great work!!1
<3
–Sam
On Thu, Jun 18, 2015 at 7:09 AM, florian.schmidt.welzow@t-online.de < florian.schmidt.welzow@t-online.de> wrote:
It's currently working via a script that you can find here: /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh
It would be great to have the script in a public version control system (e.g. github?), especially for people, e.g. volunteers, who can't ssh to gather-browser-tests.eqiad.wmflabs[1]
[1] all people, who're not members of https://wikitech.wikimedia.org/wiki/Nova_Resource:Mobile-smoketests
Best, Florian
-----Original-Nachricht----- Betreff: [WikimediaMobile] [Update] Browser tests per patch Datum: Thu, 18 Jun 2015 03:27:32 +0200 Von: Jon Robson jrobson@wikimedia.org An: "QA (software quality assurance) for Wikimedia projects." < qa@lists.wikimedia.org>, mobile-l mobile-l@lists.wikimedia.org
Background: mobile wants to gain more confidence in its browser tests by running a subset of browser tests on a case by case basis [0].
Good news: I've got a proof of concept running and Barry the browser test bot has given some legitimate helpful reviews to Gather [1].
Even better news: It's proving itself valuable already [2]. As you can see in the messages the bot has posted on [3] we have a couple of options on display option format for his reviews.
So.. hopefully this short experience has sold you all already.
This script is currently a manual job and needs a bit of tweaking before we can put it in a cron job/run it always - it needs to watch for new commits and then run a modification of the above script on a per case basis (if two versions of it run in parallel we have an issue).
Definitely something we should push for next sprint!
Long live Barry bot!
Devs... (everyone else now of what follows is likely to be useful): I got the labs instance up and running on: http://gather-browser-tests.wmflabs.org/wiki/Main_Page
Most of you in readership team should be able to ssh gather-browser-tests.eqiad.wmflabs Let me know if you have no access.
It's currently working via a script that you can find here: /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh
[0] https://phabricator.wikimedia.org/T100293 [1] https://gerrit.wikimedia.org/r/#/q/reviewer:jdlrobson%252Bbarry%2540gmail.co... [2] https://gerrit.wikimedia.org/r/#/c/218731/
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
So the script that actually runs the browser test is not in a generic useful form but it is: https://gist.github.com/jdlrobson/32b607f8009e897ee80c
It uses the GerritCommandLine tool to do grabbing and reviewing https://github.com/jdlrobson/GerritCommandLine
Ideally if we can use labs-tools-gerrit-to-redis for identifying patches and then pulling them down we wouldn't need the GerritCommandLine tool since the code to submit a review is pretty trivial and captured in this function: https://github.com/jdlrobson/GerritCommandLine/blob/master/gerrit.py#L277
I've also put this in the task https://phabricator.wikimedia.org/T101069#1379462
We'll probably want an instance per extension, to simplify having to worry about dependencies (we can just run a git update on all extensions after each checkout)
On Thu, Jun 18, 2015 at 4:32 AM, Joaquin Oltra Hernandez jhernandez@wikimedia.org wrote:
Awesome Jon! I'm so happy to finally see this developing :DD
Loving the : `Browserbot happy!`
I've noticed it can report either the name of the failing test or the full log. What do you think if we show that, and a url with the pasted log somewhere publicly to not put too much noise on the comments but still be able to see it? Something like https://phabricator.wikimedia.org/paste/...
+1 to where is the source. +1 to documenting how you've set it all up on wiki somewhere.
I also think we need a catchy phrase for the -1s!
Thanks for you work on this, we'll get more focused time for it soon.
On Thu, Jun 18, 2015 at 11:33 AM, Sam Smith samsmith@wikimedia.org wrote:
I agree with Florian everything that you've written should be in a public version control system.
Second, I'd ask that you document your experiences so far in getting this set up and how it works so that other members of the vertical can help to maintain it moving forward.
Third, great work!!1
<3
–Sam
On Thu, Jun 18, 2015 at 7:09 AM, florian.schmidt.welzow@t-online.de florian.schmidt.welzow@t-online.de wrote:
It's currently working via a script that you can find here: /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh
It would be great to have the script in a public version control system (e.g. github?), especially for people, e.g. volunteers, who can't ssh to gather-browser-tests.eqiad.wmflabs[1]
[1] all people, who're not members of https://wikitech.wikimedia.org/wiki/Nova_Resource:Mobile-smoketests
Best, Florian
-----Original-Nachricht----- Betreff: [WikimediaMobile] [Update] Browser tests per patch Datum: Thu, 18 Jun 2015 03:27:32 +0200 Von: Jon Robson jrobson@wikimedia.org An: "QA (software quality assurance) for Wikimedia projects." qa@lists.wikimedia.org, mobile-l mobile-l@lists.wikimedia.org
Background: mobile wants to gain more confidence in its browser tests by running a subset of browser tests on a case by case basis [0].
Good news: I've got a proof of concept running and Barry the browser test bot has given some legitimate helpful reviews to Gather [1].
Even better news: It's proving itself valuable already [2]. As you can see in the messages the bot has posted on [3] we have a couple of options on display option format for his reviews.
So.. hopefully this short experience has sold you all already.
This script is currently a manual job and needs a bit of tweaking before we can put it in a cron job/run it always - it needs to watch for new commits and then run a modification of the above script on a per case basis (if two versions of it run in parallel we have an issue).
Definitely something we should push for next sprint!
Long live Barry bot!
Devs... (everyone else now of what follows is likely to be useful): I got the labs instance up and running on: http://gather-browser-tests.wmflabs.org/wiki/Main_Page
Most of you in readership team should be able to ssh gather-browser-tests.eqiad.wmflabs Let me know if you have no access.
It's currently working via a script that you can find here: /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh
[0] https://phabricator.wikimedia.org/T100293 [1] https://gerrit.wikimedia.org/r/#/q/reviewer:jdlrobson%252Bbarry%2540gmail.co... [2] https://gerrit.wikimedia.org/r/#/c/218731/
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Nice work, Jon.
I've opened a task for defining a JJB builder/template and getting something like this into CI sooner rather than later.[1] I think your setup proves that a set of well-groomed MW-Selenium integration tests can be stable enough for this purpose, and we can start with an even smaller subset of core tests for a pre-merge build. Of course this isn't something that we planned to do 'now' now—sometimes 'then' suddenly becomes 'now' so 'soon'—but we can start with an experiment on Gather or MobileFrontend tests, since their health has greatly improved, and see how it goes.
[1] https://phabricator.wikimedia.org/T103039
On Thu, Jun 18, 2015 at 11:03 AM, Jon Robson jdlrobson@gmail.com wrote:
So the script that actually runs the browser test is not in a generic useful form but it is: https://gist.github.com/jdlrobson/32b607f8009e897ee80c
It uses the GerritCommandLine tool to do grabbing and reviewing https://github.com/jdlrobson/GerritCommandLine
Ideally if we can use labs-tools-gerrit-to-redis for identifying patches and then pulling them down we wouldn't need the GerritCommandLine tool since the code to submit a review is pretty trivial and captured in this function: https://github.com/jdlrobson/GerritCommandLine/blob/master/gerrit.py#L277
I've also put this in the task https://phabricator.wikimedia.org/T101069#1379462
We'll probably want an instance per extension, to simplify having to worry about dependencies (we can just run a git update on all extensions after each checkout)
On Thu, Jun 18, 2015 at 4:32 AM, Joaquin Oltra Hernandez jhernandez@wikimedia.org wrote:
Awesome Jon! I'm so happy to finally see this developing :DD
Loving the : `Browserbot happy!`
I've noticed it can report either the name of the failing test or the
full
log. What do you think if we show that, and a url with the pasted log somewhere publicly to not put too much noise on the comments but still be able to see it? Something like https://phabricator.wikimedia.org/paste/.
..
+1 to where is the source. +1 to documenting how you've set it all up on wiki somewhere.
I also think we need a catchy phrase for the -1s!
Thanks for you work on this, we'll get more focused time for it soon.
On Thu, Jun 18, 2015 at 11:33 AM, Sam Smith samsmith@wikimedia.org
wrote:
I agree with Florian everything that you've written should be in a
public
version control system.
Second, I'd ask that you document your experiences so far in getting
this
set up and how it works so that other members of the vertical can help
to
maintain it moving forward.
Third, great work!!1
<3
–Sam
On Thu, Jun 18, 2015 at 7:09 AM, florian.schmidt.welzow@t-online.de florian.schmidt.welzow@t-online.de wrote:
It's currently working via a script that you can find here: /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh
It would be great to have the script in a public version control system (e.g. github?), especially for people, e.g. volunteers, who can't ssh
to
gather-browser-tests.eqiad.wmflabs[1]
[1] all people, who're not members of https://wikitech.wikimedia.org/wiki/Nova_Resource:Mobile-smoketests
Best, Florian
-----Original-Nachricht----- Betreff: [WikimediaMobile] [Update] Browser tests per patch Datum: Thu, 18 Jun 2015 03:27:32 +0200 Von: Jon Robson jrobson@wikimedia.org An: "QA (software quality assurance) for Wikimedia projects." qa@lists.wikimedia.org, mobile-l mobile-l@lists.wikimedia.org
Background: mobile wants to gain more confidence in its browser tests by running a subset of browser tests on a case by case basis [0].
Good news: I've got a proof of concept running and Barry the browser test bot has given some legitimate helpful reviews to Gather [1].
Even better news: It's proving itself valuable already [2]. As you can see in the messages the bot has posted on [3] we have a couple of options on display option format for his reviews.
So.. hopefully this short experience has sold you all already.
This script is currently a manual job and needs a bit of tweaking before we can put it in a cron job/run it always - it needs to watch for new commits and then run a modification of the above script on a per case basis (if two versions of it run in parallel we have an issue).
Definitely something we should push for next sprint!
Long live Barry bot!
Devs... (everyone else now of what follows is likely to be useful): I got the labs instance up and running on: http://gather-browser-tests.wmflabs.org/wiki/Main_Page
Most of you in readership team should be able to ssh gather-browser-tests.eqiad.wmflabs Let me know if you have no access.
It's currently working via a script that you can find here: /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh
https://gerrit.wikimedia.org/r/#/q/reviewer:jdlrobson%252Bbarry%2540gmail.co...
[2] https://gerrit.wikimedia.org/r/#/c/218731/
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Jon Robson
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
So far I'm seeing some extremely positive results. The tests are running super fast (in phantomjs the smoke tests are taking less than 3 minutes...).
I've been manually running him whilst I do code review in parallel and he's already generated some interesting conversation on this patchset: https://gerrit.wikimedia.org/r/#/c/219249/
If you want to pair and get this to be less hacky I'm more than happy!
On Thu, Jun 18, 2015 at 5:41 PM, Dan Duvall dduvall@wikimedia.org wrote:
Nice work, Jon.
I've opened a task for defining a JJB builder/template and getting something like this into CI sooner rather than later.[1] I think your setup proves that a set of well-groomed MW-Selenium integration tests can be stable enough for this purpose, and we can start with an even smaller subset of core tests for a pre-merge build. Of course this isn't something that we planned to do 'now' now—sometimes 'then' suddenly becomes 'now' so 'soon'—but we can start with an experiment on Gather or MobileFrontend tests, since their health has greatly improved, and see how it goes.
[1] https://phabricator.wikimedia.org/T103039
On Thu, Jun 18, 2015 at 11:03 AM, Jon Robson jdlrobson@gmail.com wrote:
So the script that actually runs the browser test is not in a generic useful form but it is: https://gist.github.com/jdlrobson/32b607f8009e897ee80c
It uses the GerritCommandLine tool to do grabbing and reviewing https://github.com/jdlrobson/GerritCommandLine
Ideally if we can use labs-tools-gerrit-to-redis for identifying patches and then pulling them down we wouldn't need the GerritCommandLine tool since the code to submit a review is pretty trivial and captured in this function: https://github.com/jdlrobson/GerritCommandLine/blob/master/gerrit.py#L277
I've also put this in the task https://phabricator.wikimedia.org/T101069#1379462
We'll probably want an instance per extension, to simplify having to worry about dependencies (we can just run a git update on all extensions after each checkout)
On Thu, Jun 18, 2015 at 4:32 AM, Joaquin Oltra Hernandez jhernandez@wikimedia.org wrote:
Awesome Jon! I'm so happy to finally see this developing :DD
Loving the : `Browserbot happy!`
I've noticed it can report either the name of the failing test or the full log. What do you think if we show that, and a url with the pasted log somewhere publicly to not put too much noise on the comments but still be able to see it? Something like https://phabricator.wikimedia.org/paste/...
+1 to where is the source. +1 to documenting how you've set it all up on wiki somewhere.
I also think we need a catchy phrase for the -1s!
Thanks for you work on this, we'll get more focused time for it soon.
On Thu, Jun 18, 2015 at 11:33 AM, Sam Smith samsmith@wikimedia.org wrote:
I agree with Florian everything that you've written should be in a public version control system.
Second, I'd ask that you document your experiences so far in getting this set up and how it works so that other members of the vertical can help to maintain it moving forward.
Third, great work!!1
<3
–Sam
On Thu, Jun 18, 2015 at 7:09 AM, florian.schmidt.welzow@t-online.de florian.schmidt.welzow@t-online.de wrote:
It's currently working via a script that you can find here: /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh
It would be great to have the script in a public version control system (e.g. github?), especially for people, e.g. volunteers, who can't ssh to gather-browser-tests.eqiad.wmflabs[1]
[1] all people, who're not members of https://wikitech.wikimedia.org/wiki/Nova_Resource:Mobile-smoketests
Best, Florian
-----Original-Nachricht----- Betreff: [WikimediaMobile] [Update] Browser tests per patch Datum: Thu, 18 Jun 2015 03:27:32 +0200 Von: Jon Robson jrobson@wikimedia.org An: "QA (software quality assurance) for Wikimedia projects." qa@lists.wikimedia.org, mobile-l mobile-l@lists.wikimedia.org
Background: mobile wants to gain more confidence in its browser tests by running a subset of browser tests on a case by case basis [0].
Good news: I've got a proof of concept running and Barry the browser test bot has given some legitimate helpful reviews to Gather [1].
Even better news: It's proving itself valuable already [2]. As you can see in the messages the bot has posted on [3] we have a couple of options on display option format for his reviews.
So.. hopefully this short experience has sold you all already.
This script is currently a manual job and needs a bit of tweaking before we can put it in a cron job/run it always - it needs to watch for new commits and then run a modification of the above script on a per case basis (if two versions of it run in parallel we have an issue).
Definitely something we should push for next sprint!
Long live Barry bot!
Devs... (everyone else now of what follows is likely to be useful): I got the labs instance up and running on: http://gather-browser-tests.wmflabs.org/wiki/Main_Page
Most of you in readership team should be able to ssh gather-browser-tests.eqiad.wmflabs Let me know if you have no access.
It's currently working via a script that you can find here: /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh
[0] https://phabricator.wikimedia.org/T100293 [1]
https://gerrit.wikimedia.org/r/#/q/reviewer:jdlrobson%252Bbarry%2540gmail.co... [2] https://gerrit.wikimedia.org/r/#/c/218731/
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Jon Robson
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Dan Duvall Automation Engineer Wikimedia Foundation
May I suggest that Barry should not +1 patches. It gives a false impression that the code is alright even though the code may not be covered at all. Can we have it just -1 when there is a problem, and stay silent otherwise?
On Jun 18, 2015, at 8:55 PM, Jon Robson jdlrobson@gmail.com wrote:
So far I'm seeing some extremely positive results. The tests are running super fast (in phantomjs the smoke tests are taking less than 3 minutes...).
I've been manually running him whilst I do code review in parallel and he's already generated some interesting conversation on this patchset: https://gerrit.wikimedia.org/r/#/c/219249/ https://gerrit.wikimedia.org/r/#/c/219249/
If you want to pair and get this to be less hacky I'm more than happy!
On Thu, Jun 18, 2015 at 5:41 PM, Dan Duvall <dduvall@wikimedia.org mailto:dduvall@wikimedia.org> wrote:
Nice work, Jon.
I've opened a task for defining a JJB builder/template and getting something like this into CI sooner rather than later.[1] I think your setup proves that a set of well-groomed MW-Selenium integration tests can be stable enough for this purpose, and we can start with an even smaller subset of core tests for a pre-merge build. Of course this isn't something that we planned to do 'now' now—sometimes 'then' suddenly becomes 'now' so 'soon'—but we can start with an experiment on Gather or MobileFrontend tests, since their health has greatly improved, and see how it goes.
[1] https://phabricator.wikimedia.org/T103039
On Thu, Jun 18, 2015 at 11:03 AM, Jon Robson jdlrobson@gmail.com wrote:
So the script that actually runs the browser test is not in a generic useful form but it is: https://gist.github.com/jdlrobson/32b607f8009e897ee80c
It uses the GerritCommandLine tool to do grabbing and reviewing https://github.com/jdlrobson/GerritCommandLine
Ideally if we can use labs-tools-gerrit-to-redis for identifying patches and then pulling them down we wouldn't need the GerritCommandLine tool since the code to submit a review is pretty trivial and captured in this function: https://github.com/jdlrobson/GerritCommandLine/blob/master/gerrit.py#L277
I've also put this in the task https://phabricator.wikimedia.org/T101069#1379462
We'll probably want an instance per extension, to simplify having to worry about dependencies (we can just run a git update on all extensions after each checkout)
On Thu, Jun 18, 2015 at 4:32 AM, Joaquin Oltra Hernandez jhernandez@wikimedia.org wrote:
Awesome Jon! I'm so happy to finally see this developing :DD
Loving the : `Browserbot happy!`
I've noticed it can report either the name of the failing test or the full log. What do you think if we show that, and a url with the pasted log somewhere publicly to not put too much noise on the comments but still be able to see it? Something like https://phabricator.wikimedia.org/paste/...
+1 to where is the source. +1 to documenting how you've set it all up on wiki somewhere.
I also think we need a catchy phrase for the -1s!
Thanks for you work on this, we'll get more focused time for it soon.
On Thu, Jun 18, 2015 at 11:33 AM, Sam Smith samsmith@wikimedia.org wrote:
I agree with Florian everything that you've written should be in a public version control system.
Second, I'd ask that you document your experiences so far in getting this set up and how it works so that other members of the vertical can help to maintain it moving forward.
Third, great work!!1
<3
–Sam
On Thu, Jun 18, 2015 at 7:09 AM, florian.schmidt.welzow@t-online.de florian.schmidt.welzow@t-online.de wrote:
> It's currently working via a script that you can find here: > /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh
It would be great to have the script in a public version control system (e.g. github?), especially for people, e.g. volunteers, who can't ssh to gather-browser-tests.eqiad.wmflabs[1]
[1] all people, who're not members of https://wikitech.wikimedia.org/wiki/Nova_Resource:Mobile-smoketests
Best, Florian
-----Original-Nachricht----- Betreff: [WikimediaMobile] [Update] Browser tests per patch Datum: Thu, 18 Jun 2015 03:27:32 +0200 Von: Jon Robson jrobson@wikimedia.org An: "QA (software quality assurance) for Wikimedia projects." qa@lists.wikimedia.org, mobile-l mobile-l@lists.wikimedia.org
Background: mobile wants to gain more confidence in its browser tests by running a subset of browser tests on a case by case basis [0].
Good news: I've got a proof of concept running and Barry the browser test bot has given some legitimate helpful reviews to Gather [1].
Even better news: It's proving itself valuable already [2]. As you can see in the messages the bot has posted on [3] we have a couple of options on display option format for his reviews.
So.. hopefully this short experience has sold you all already.
This script is currently a manual job and needs a bit of tweaking before we can put it in a cron job/run it always - it needs to watch for new commits and then run a modification of the above script on a per case basis (if two versions of it run in parallel we have an issue).
Definitely something we should push for next sprint!
Long live Barry bot!
Devs... (everyone else now of what follows is likely to be useful): I got the labs instance up and running on: http://gather-browser-tests.wmflabs.org/wiki/Main_Page
Most of you in readership team should be able to ssh gather-browser-tests.eqiad.wmflabs Let me know if you have no access.
It's currently working via a script that you can find here: /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh
[0] https://phabricator.wikimedia.org/T100293 [1]
https://gerrit.wikimedia.org/r/#/q/reviewer:jdlrobson%252Bbarry%2540gmail.co... [2] https://gerrit.wikimedia.org/r/#/c/218731/
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Jon Robson
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Dan Duvall Automation Engineer Wikimedia Foundation
-- Jon Robson
- http://jonrobson.me.uk http://jonrobson.me.uk/
- https://www.facebook.com/jonrobson https://www.facebook.com/jonrobson
- @rakugojon
Mobile-l mailing list Mobile-l@lists.wikimedia.org mailto:Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l https://lists.wikimedia.org/mailman/listinfo/mobile-l
Or, maybe, it should be possible for the bot to set the verified flag, instead of the code review (it's not a code review, so a minus one is misleading, too).
Freundliche Grüße Florian Schmidt
-----Original-Nachricht-----
Betreff: Re: [WikimediaMobile] [Update] Browser tests per patch
Datum: Wed, 24 Jun 2015 13:47:39 +0200
Von: Bahodir Mansurov bmansurov@wikimedia.org
An: Jon Robson jdlrobson@gmail.com
May I suggest that Barry should not +1 patches. It gives a false impression that the code is alright even though the code may not be covered at all. Can we have it just -1 when there is a problem, and stay silent otherwise? On Jun 18, 2015, at 8:55 PM, Jon Robson <jdlrobson@gmail.com [1]> wrote: So far I'm seeing some extremely positive results. The tests are running super fast (in phantomjs the smoke tests are taking less than 3 minutes...).
I've been manually running him whilst I do code review in parallel and he's already generated some interesting conversation on this patchset: https://gerrit.wikimedia.org/r/#/c/219249/ [2]
If you want to pair and get this to be less hacky I'm more than happy!
On Thu, Jun 18, 2015 at 5:41 PM, Dan Duvall <dduvall@wikimedia.org [3]> wrote: Nice work, Jon.
I've opened a task for defining a JJB builder/template and getting something like this into CI sooner rather than later.[1] I think your setup proves that a set of well-groomed MW-Selenium integration tests can be stable enough for this purpose, and we can start with an even smaller subset of core tests for a pre-merge build. Of course this isn't something that we planned to do 'now' now—sometimes 'then' suddenly becomes 'now' so 'soon'—but we can start with an experiment on Gather or MobileFrontend tests, since their health has greatly improved, and see how it goes.
[1] https://phabricator.wikimedia.org/T103039 [4]
On Thu, Jun 18, 2015 at 11:03 AM, Jon Robson <jdlrobson@gmail.com [5]> wrote:
So the script that actually runs the browser test is not in a generic useful form but it is: https://gist.github.com/jdlrobson/32b607f8009e897ee80c [6]
It uses the GerritCommandLine tool to do grabbing and reviewing https://github.com/jdlrobson/GerritCommandLine
Ideally if we can use labs-tools-gerrit-to-redis for identifying patches and then pulling them down we wouldn't need the GerritCommandLine tool since the code to submit a review is pretty trivial and captured in this function: https://github.com/jdlrobson/GerritCommandLine/blob/master/gerrit.py#L277
I've also put this in the task https://phabricator.wikimedia.org/T101069#1379462
We'll probably want an instance per extension, to simplify having to worry about dependencies (we can just run a git update on all extensions after each checkout)
On Thu, Jun 18, 2015 at 4:32 AM, Joaquin Oltra Hernandez jhernandez@wikimedia.org wrote: Awesome Jon! I'm so happy to finally see this developing :DD
Loving the : `Browserbot happy!`
I've noticed it can report either the name of the failing test or the full log. What do you think if we show that, and a url with the pasted log somewhere publicly to not put too much noise on the comments but still be able to see it? Something like https://phabricator.wikimedia.org/paste/...
+1 to where is the source. +1 to documenting how you've set it all up on wiki somewhere.
I also think we need a catchy phrase for the -1s!
Thanks for you work on this, we'll get more focused time for it soon.
On Thu, Jun 18, 2015 at 11:33 AM, Sam Smith samsmith@wikimedia.org wrote:
I agree with Florian everything that you've written should be in a public version control system.
Second, I'd ask that you document your experiences so far in getting this set up and how it works so that other members of the vertical can help to maintain it moving forward.
Third, great work!!1
<3
–Sam
On Thu, Jun 18, 2015 at 7:09 AM, florian.schmidt.welzow@t-online.de florian.schmidt.welzow@t-online.de wrote:
It's currently working via a script that you can find here: /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh It would be great to have the script in a public version control system (e.g. github?), especially for people, e.g. volunteers, who can't ssh to gather-browser-tests.eqiad.wmflabs[1]
[1] all people, who're not members of https://wikitech.wikimedia.org/wiki/Nova_Resource:Mobile-smoketests
Best, Florian
-----Original-Nachricht----- Betreff: [WikimediaMobile] [Update] Browser tests per patch Datum: Thu, 18 Jun 2015 03:27:32 +0200 Von: Jon Robson jrobson@wikimedia.org An: "QA (software quality assurance) for Wikimedia projects." qa@lists.wikimedia.org, mobile-l mobile-l@lists.wikimedia.org
Background: mobile wants to gain more confidence in its browser tests by running a subset of browser tests on a case by case basis [0].
Good news: I've got a proof of concept running and Barry the browser test bot has given some legitimate helpful reviews to Gather [1].
Even better news: It's proving itself valuable already [2]. As you can see in the messages the bot has posted on [3] we have a couple of options on display option format for his reviews.
So.. hopefully this short experience has sold you all already.
This script is currently a manual job and needs a bit of tweaking before we can put it in a cron job/run it always - it needs to watch for new commits and then run a modification of the above script on a per case basis (if two versions of it run in parallel we have an issue).
Definitely something we should push for next sprint!
Long live Barry bot!
Devs... (everyone else now of what follows is likely to be useful): I got the labs instance up and running on: http://gather-browser-tests.wmflabs.org/wiki/Main_Page
Most of you in readership team should be able to ssh gather-browser-tests.eqiad.wmflabs Let me know if you have no access.
It's currently working via a script that you can find here: /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh
[0] https://phabricator.wikimedia.org/T100293 [1]
https://gerrit.wikimedia.org/r/#/q/reviewer:jdlrobson%252Bbarry%2540gmail.co... [2] https://gerrit.wikimedia.org/r/#/c/218731/
_______________________________________________ Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
_______________________________________________ Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
_______________________________________________ Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
_______________________________________________ Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Jon Robson * http://jonrobson.me.uk * https://www.facebook.com/jonrobson * @rakugojon
_______________________________________________ Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Dan Duvall Automation Engineer Wikimedia Foundation
Baha I agree with you, but instead of staying silent just commenting a "BarryBot is happy! Browser tests passed" without setting any +1 would be good to know they run.
I also agree with Florian, if we could set Verified +1 or -1 that could be interesting. How does it affect jenkins if something/somebody sets verified to -1 for example? Would it block it from merging?
On Wed, Jun 24, 2015 at 1:53 PM, florian.schmidt.welzow@t-online.de < florian.schmidt.welzow@t-online.de> wrote:
Or, maybe, it should be possible for the bot to set the verified flag, instead of the code review (it's not a code review, so a minus one is misleading, too).
Freundliche Grüße Florian Schmidt
-----Original-Nachricht-----
Betreff: Re: [WikimediaMobile] [Update] Browser tests per patch
Datum: Wed, 24 Jun 2015 13:47:39 +0200
Von: Bahodir Mansurov bmansurov@wikimedia.org
An: Jon Robson jdlrobson@gmail.com
May I suggest that Barry should not +1 patches. It gives a false impression that the code is alright even though the code may not be covered at all. Can we have it just -1 when there is a problem, and stay silent otherwise?
On Jun 18, 2015, at 8:55 PM, Jon Robson jdlrobson@gmail.com wrote:
So far I'm seeing some extremely positive results. The tests are running super fast (in phantomjs the smoke tests are taking less than 3 minutes...).
I've been manually running him whilst I do code review in parallel and he's already generated some interesting conversation on this patchset: https://gerrit.wikimedia.org/r/#/c/219249/
If you want to pair and get this to be less hacky I'm more than happy!
On Thu, Jun 18, 2015 at 5:41 PM, Dan Duvall dduvall@wikimedia.org wrote:
Nice work, Jon.
I've opened a task for defining a JJB builder/template and getting something like this into CI sooner rather than later.[1] I think your setup proves that a set of well-groomed MW-Selenium integration tests can be stable enough for this purpose, and we can start with an even smaller subset of core tests for a pre-merge build. Of course this isn't something that we planned to do 'now' now—sometimes 'then' suddenly becomes 'now' so 'soon'—but we can start with an experiment on Gather or MobileFrontend tests, since their health has greatly improved, and see how it goes.
[1] https://phabricator.wikimedia.org/T103039
On Thu, Jun 18, 2015 at 11:03 AM, Jon Robson jdlrobson@gmail.com wrote:
So the script that actually runs the browser test is not in a generic useful form but it is: https://gist.github.com/jdlrobson/32b607f8009e897ee80c
It uses the GerritCommandLine tool to do grabbing and reviewing https://github.com/jdlrobson/GerritCommandLine
Ideally if we can use labs-tools-gerrit-to-redis for identifying patches and then pulling them down we wouldn't need the GerritCommandLine tool since the code to submit a review is pretty trivial and captured in this function: https://github.com/jdlrobson/GerritCommandLine/blob/master/gerrit.py#L277
I've also put this in the task https://phabricator.wikimedia.org/T101069#1379462
We'll probably want an instance per extension, to simplify having to worry about dependencies (we can just run a git update on all extensions after each checkout)
On Thu, Jun 18, 2015 at 4:32 AM, Joaquin Oltra Hernandez jhernandez@wikimedia.org wrote:
Awesome Jon! I'm so happy to finally see this developing :DD
Loving the : `Browserbot happy!`
I've noticed it can report either the name of the failing test or the full log. What do you think if we show that, and a url with the pasted log somewhere publicly to not put too much noise on the comments but still be able to see it? Something like https://phabricator.wikimedia.org/paste/...
+1 to where is the source. +1 to documenting how you've set it all up on wiki somewhere.
I also think we need a catchy phrase for the -1s!
Thanks for you work on this, we'll get more focused time for it soon.
On Thu, Jun 18, 2015 at 11:33 AM, Sam Smith samsmith@wikimedia.org wrote:
I agree with Florian everything that you've written should be in a public version control system.
Second, I'd ask that you document your experiences so far in getting this set up and how it works so that other members of the vertical can help to maintain it moving forward.
Third, great work!!1
<3
–Sam
On Thu, Jun 18, 2015 at 7:09 AM, florian.schmidt.welzow@t-online.de florian.schmidt.welzow@t-online.de wrote:
It's currently working via a script that you can find here: /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh
It would be great to have the script in a public version control system (e.g. github?), especially for people, e.g. volunteers, who can't ssh to gather-browser-tests.eqiad.wmflabs[1]
[1] all people, who're not members of https://wikitech.wikimedia.org/wiki/Nova_Resource:Mobile-smoketests
Best, Florian
-----Original-Nachricht----- Betreff: [WikimediaMobile] [Update] Browser tests per patch Datum: Thu, 18 Jun 2015 03:27:32 +0200 Von: Jon Robson jrobson@wikimedia.org An: "QA (software quality assurance) for Wikimedia projects." qa@lists.wikimedia.org, mobile-l mobile-l@lists.wikimedia.org
Background: mobile wants to gain more confidence in its browser tests by running a subset of browser tests on a case by case basis [0].
Good news: I've got a proof of concept running and Barry the browser test bot has given some legitimate helpful reviews to Gather [1].
Even better news: It's proving itself valuable already [2]. As you can see in the messages the bot has posted on [3] we have a couple of options on display option format for his reviews.
So.. hopefully this short experience has sold you all already.
This script is currently a manual job and needs a bit of tweaking before we can put it in a cron job/run it always - it needs to watch for new commits and then run a modification of the above script on a per case basis (if two versions of it run in parallel we have an issue).
Definitely something we should push for next sprint!
Long live Barry bot!
Devs... (everyone else now of what follows is likely to be useful): I got the labs instance up and running on: http://gather-browser-tests.wmflabs.org/wiki/Main_Page
Most of you in readership team should be able to ssh gather-browser-tests.eqiad.wmflabs Let me know if you have no access.
It's currently working via a script that you can find here: /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh
[0] https://phabricator.wikimedia.org/T100293 [1]
https://gerrit.wikimedia.org/r/#/q/reviewer:jdlrobson%252Bbarry%2540gmail.co... [2] https://gerrit.wikimedia.org/r/#/c/218731/
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Jon Robson
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Dan Duvall Automation Engineer Wikimedia Foundation
-- Jon Robson
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
How does it affect jenkins if something/somebody sets verified to -1
for example? Would it block it from merging?
Any -1/-2 in the verified column blocks submitting a change (even, if other reviewers add a +1 in verified), so if Barry wouldn't be happy, you can't merge. A workaround is to remove BarryBot as a reviewer (which would remove the vote, too, and allows jenkins to submit the change).
Freundliche Grüße Florian Schmidt
-----Original-Nachricht-----
Betreff: Re: [WikimediaMobile] [Update] Browser tests per patch
Datum: Wed, 24 Jun 2015 14:07:23 +0200
Von: Joaquin Oltra Hernandez jhernandez@wikimedia.org
An: "florian.schmidt.welzow@t-online.de" florian.schmidt.welzow@t-online.de
Baha I agree with you, but instead of staying silent just commenting a "BarryBot is happy! Browser tests passed" without setting any +1 would be good to know they run. I also agree with Florian, if we could set Verified +1 or -1 that could be interesting. How does it affect jenkins if something/somebody sets verified to -1 for example? Would it block it from merging? On Wed, Jun 24, 2015 at 1:53 PM, florian.schmidt.welzow@t-online.de [1] <florian.schmidt.welzow@t-online.de [2]> wrote:
Or, maybe, it should be possible for the bot to set the verified flag, instead of the code review (it's not a code review, so a minus one is misleading, too).
Freundliche Grüße Florian Schmidt
-----Original-Nachricht-----
Betreff: Re: [WikimediaMobile] [Update] Browser tests per patch
Datum: Wed, 24 Jun 2015 13:47:39 +0200
Von: Bahodir Mansurov <bmansurov@wikimedia.org [3]>
An: Jon Robson <jdlrobson@gmail.com [4]>
May I suggest that Barry should not +1 patches. It gives a false impression that the code is alright even though the code may not be covered at all. Can we have it just -1 when there is a problem, and stay silent otherwise? On Jun 18, 2015, at 8:55 PM, Jon Robson <jdlrobson@gmail.com [5]> wrote: So far I'm seeing some extremely positive results. The tests are running super fast (in phantomjs the smoke tests are taking less than 3 minutes...).
I've been manually running him whilst I do code review in parallel and he's already generated some interesting conversation on this patchset: https://gerrit.wikimedia.org/r/#/c/219249/ [6]
If you want to pair and get this to be less hacky I'm more than happy!
On Thu, Jun 18, 2015 at 5:41 PM, Dan Duvall <dduvall@wikimedia.org [7]> wrote: Nice work, Jon.
I've opened a task for defining a JJB builder/template and getting something like this into CI sooner rather than later.[1] I think your setup proves that a set of well-groomed MW-Selenium integration tests can be stable enough for this purpose, and we can start with an even smaller subset of core tests for a pre-merge build. Of course this isn't something that we planned to do 'now' now—sometimes 'then' suddenly becomes 'now' so 'soon'—but we can start with an experiment on Gather or MobileFrontend tests, since their health has greatly improved, and see how it goes.
[1] https://phabricator.wikimedia.org/T103039 [8]
On Thu, Jun 18, 2015 at 11:03 AM, Jon Robson <jdlrobson@gmail.com [9]> wrote:
So the script that actually runs the browser test is not in a generic useful form but it is: https://gist.github.com/jdlrobson/32b607f8009e897ee80c [10]
It uses the GerritCommandLine tool to do grabbing and reviewing https://github.com/jdlrobson/GerritCommandLine [11]
Ideally if we can use labs-tools-gerrit-to-redis for identifying patches and then pulling them down we wouldn't need the GerritCommandLine tool since the code to submit a review is pretty trivial and captured in this function: https://github.com/jdlrobson/GerritCommandLine/blob/master/gerrit.py#L277 [12]
I've also put this in the task https://phabricator.wikimedia.org/T101069#1379462 [13]
We'll probably want an instance per extension, to simplify having to worry about dependencies (we can just run a git update on all extensions after each checkout)
On Thu, Jun 18, 2015 at 4:32 AM, Joaquin Oltra Hernandez <jhernandez@wikimedia.org [14]> wrote: Awesome Jon! I'm so happy to finally see this developing :DD
Loving the : `Browserbot happy!`
I've noticed it can report either the name of the failing test or the full log. What do you think if we show that, and a url with the pasted log somewhere publicly to not put too much noise on the comments but still be able to see it? Something like https://phabricator.wikimedia.org/paste/. [15]..
+1 to where is the source. +1 to documenting how you've set it all up on wiki somewhere.
I also think we need a catchy phrase for the -1s!
Thanks for you work on this, we'll get more focused time for it soon.
On Thu, Jun 18, 2015 at 11:33 AM, Sam Smith <samsmith@wikimedia.org [16]> wrote:
I agree with Florian everything that you've written should be in a public version control system.
Second, I'd ask that you document your experiences so far in getting this set up and how it works so that other members of the vertical can help to maintain it moving forward.
Third, great work!!1
<3
–Sam
On Thu, Jun 18, 2015 at 7:09 AM, florian.schmidt.welzow@t-online.de [17] <florian.schmidt.welzow@t-online.de [18]> wrote:
It's currently working via a script that you can find here: /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh It would be great to have the script in a public version control system (e.g. github?), especially for people, e.g. volunteers, who can't ssh to gather-browser-tests.eqiad.wmflabs[1]
[1] all people, who're not members of https://wikitech.wikimedia.org/wiki/Nova_Resource:Mobile-smoketests [19]
Best, Florian
-----Original-Nachricht----- Betreff: [WikimediaMobile] [Update] Browser tests per patch Datum: Thu, 18 Jun 2015 03:27:32 +0200 Von: Jon Robson <jrobson@wikimedia.org [20]> An: "QA (software quality assurance) for Wikimedia projects." <qa@lists.wikimedia.org [21]>, mobile-l <mobile-l@lists.wikimedia.org [22]>
Background: mobile wants to gain more confidence in its browser tests by running a subset of browser tests on a case by case basis [0].
Good news: I've got a proof of concept running and Barry the browser test bot has given some legitimate helpful reviews to Gather [1].
Even better news: It's proving itself valuable already [2]. As you can see in the messages the bot has posted on [3] we have a couple of options on display option format for his reviews.
So.. hopefully this short experience has sold you all already.
This script is currently a manual job and needs a bit of tweaking before we can put it in a cron job/run it always - it needs to watch for new commits and then run a modification of the above script on a per case basis (if two versions of it run in parallel we have an issue).
Definitely something we should push for next sprint!
Long live Barry bot!
Devs... (everyone else now of what follows is likely to be useful): I got the labs instance up and running on: http://gather-browser-tests.wmflabs.org/wiki/Main_Page [23]
Most of you in readership team should be able to ssh gather-browser-tests.eqiad.wmflabs Let me know if you have no access.
It's currently working via a script that you can find here: /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh
[0] https://phabricator.wikimedia.org/T100293 [24] [1]
https://gerrit.wikimedia.org/r/#/q/reviewer:jdlrobson%252Bbarry%2540gmail.co... [25] [2] https://gerrit.wikimedia.org/r/#/c/218731/ [26]
_______________________________________________ Mobile-l mailing list Mobile-l@lists.wikimedia.org [27] https://lists.wikimedia.org/mailman/listinfo/mobile-l [28]
_______________________________________________ Mobile-l mailing list Mobile-l@lists.wikimedia.org [29] https://lists.wikimedia.org/mailman/listinfo/mobile-l [30]
_______________________________________________ Mobile-l mailing list Mobile-l@lists.wikimedia.org [31] https://lists.wikimedia.org/mailman/listinfo/mobile-l [32]
_______________________________________________ Mobile-l mailing list Mobile-l@lists.wikimedia.org [33] https://lists.wikimedia.org/mailman/listinfo/mobile-l [34]
-- Jon Robson * http://jonrobson.me.uk [35] * https://www.facebook.com/jonrobson [36] * @rakugojon
_______________________________________________ Mobile-l mailing list Mobile-l@lists.wikimedia.org [37] https://lists.wikimedia.org/mailman/listinfo/mobile-l [38]
-- Dan Duvall Automation Engineer Wikimedia Foundation
I haven't been able to use verified yet - I think this is a permission that Barry doesn't have. I'm experimenting with him doing no score reviews right now rather than +1s for good patches. The -1s are worth being as visible as possible.
I'm hoping to send an update on the status quo but I've updated him so that he can be run in a script: https://github.com/jdlrobson/Barry-the-Browser-Test-Bot/blob/master/barrybot... If you want him to re-review code simply remove him from the list of reviewers - the bot currently works out what it needs to review based on which patches he is currently not a reviewer for.
On Wed, Jun 24, 2015 at 5:36 AM, florian.schmidt.welzow@t-online.de florian.schmidt.welzow@t-online.de wrote:
How does it affect jenkins if something/somebody sets verified to -1 for example? Would it block it from merging?
Any -1/-2 in the verified column blocks submitting a change (even, if other reviewers add a +1 in verified), so if Barry wouldn't be happy, you can't merge. A workaround is to remove BarryBot as a reviewer (which would remove the vote, too, and allows jenkins to submit the change).
Freundliche Grüße Florian Schmidt
-----Original-Nachricht-----
Betreff: Re: [WikimediaMobile] [Update] Browser tests per patch
Datum: Wed, 24 Jun 2015 14:07:23 +0200
Von: Joaquin Oltra Hernandez jhernandez@wikimedia.org
An: "florian.schmidt.welzow@t-online.de" florian.schmidt.welzow@t-online.de
Baha I agree with you, but instead of staying silent just commenting a "BarryBot is happy! Browser tests passed" without setting any +1 would be good to know they run.
I also agree with Florian, if we could set Verified +1 or -1 that could be interesting. How does it affect jenkins if something/somebody sets verified to -1 for example? Would it block it from merging?
On Wed, Jun 24, 2015 at 1:53 PM, florian.schmidt.welzow@t-online.de florian.schmidt.welzow@t-online.de wrote:
Or, maybe, it should be possible for the bot to set the verified flag, instead of the code review (it's not a code review, so a minus one is misleading, too).
Freundliche Grüße Florian Schmidt
-----Original-Nachricht-----
Betreff: Re: [WikimediaMobile] [Update] Browser tests per patch
Datum: Wed, 24 Jun 2015 13:47:39 +0200
Von: Bahodir Mansurov bmansurov@wikimedia.org
An: Jon Robson jdlrobson@gmail.com
May I suggest that Barry should not +1 patches. It gives a false impression that the code is alright even though the code may not be covered at all. Can we have it just -1 when there is a problem, and stay silent otherwise?
On Jun 18, 2015, at 8:55 PM, Jon Robson jdlrobson@gmail.com wrote:
So far I'm seeing some extremely positive results. The tests are running super fast (in phantomjs the smoke tests are taking less than 3 minutes...).
I've been manually running him whilst I do code review in parallel and he's already generated some interesting conversation on this patchset: https://gerrit.wikimedia.org/r/#/c/219249/
If you want to pair and get this to be less hacky I'm more than happy!
On Thu, Jun 18, 2015 at 5:41 PM, Dan Duvall dduvall@wikimedia.org wrote:
Nice work, Jon.
I've opened a task for defining a JJB builder/template and getting something like this into CI sooner rather than later.[1] I think your setup proves that a set of well-groomed MW-Selenium integration tests can be stable enough for this purpose, and we can start with an even smaller subset of core tests for a pre-merge build. Of course this isn't something that we planned to do 'now' now—sometimes 'then' suddenly becomes 'now' so 'soon'—but we can start with an experiment on Gather or MobileFrontend tests, since their health has greatly improved, and see how it goes.
[1] https://phabricator.wikimedia.org/T103039
On Thu, Jun 18, 2015 at 11:03 AM, Jon Robson jdlrobson@gmail.com wrote:
So the script that actually runs the browser test is not in a generic useful form but it is: https://gist.github.com/jdlrobson/32b607f8009e897ee80c
It uses the GerritCommandLine tool to do grabbing and reviewing https://github.com/jdlrobson/GerritCommandLine
Ideally if we can use labs-tools-gerrit-to-redis for identifying patches and then pulling them down we wouldn't need the GerritCommandLine tool since the code to submit a review is pretty trivial and captured in this function: https://github.com/jdlrobson/GerritCommandLine/blob/master/gerrit.py#L277
I've also put this in the task https://phabricator.wikimedia.org/T101069#1379462
We'll probably want an instance per extension, to simplify having to worry about dependencies (we can just run a git update on all extensions after each checkout)
On Thu, Jun 18, 2015 at 4:32 AM, Joaquin Oltra Hernandez jhernandez@wikimedia.org wrote:
Awesome Jon! I'm so happy to finally see this developing :DD
Loving the : `Browserbot happy!`
I've noticed it can report either the name of the failing test or the full log. What do you think if we show that, and a url with the pasted log somewhere publicly to not put too much noise on the comments but still be able to see it? Something like https://phabricator.wikimedia.org/paste/...
+1 to where is the source. +1 to documenting how you've set it all up on wiki somewhere.
I also think we need a catchy phrase for the -1s!
Thanks for you work on this, we'll get more focused time for it soon.
On Thu, Jun 18, 2015 at 11:33 AM, Sam Smith samsmith@wikimedia.org wrote:
I agree with Florian everything that you've written should be in a public version control system.
Second, I'd ask that you document your experiences so far in getting this set up and how it works so that other members of the vertical can help to maintain it moving forward.
Third, great work!!1
<3
–Sam
On Thu, Jun 18, 2015 at 7:09 AM, florian.schmidt.welzow@t-online.de florian.schmidt.welzow@t-online.de wrote:
It's currently working via a script that you can find here: /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh
It would be great to have the script in a public version control system (e.g. github?), especially for people, e.g. volunteers, who can't ssh to gather-browser-tests.eqiad.wmflabs[1]
[1] all people, who're not members of https://wikitech.wikimedia.org/wiki/Nova_Resource:Mobile-smoketests
Best, Florian
-----Original-Nachricht----- Betreff: [WikimediaMobile] [Update] Browser tests per patch Datum: Thu, 18 Jun 2015 03:27:32 +0200 Von: Jon Robson jrobson@wikimedia.org An: "QA (software quality assurance) for Wikimedia projects." qa@lists.wikimedia.org, mobile-l mobile-l@lists.wikimedia.org
Background: mobile wants to gain more confidence in its browser tests by running a subset of browser tests on a case by case basis [0].
Good news: I've got a proof of concept running and Barry the browser test bot has given some legitimate helpful reviews to Gather [1].
Even better news: It's proving itself valuable already [2]. As you can see in the messages the bot has posted on [3] we have a couple of options on display option format for his reviews.
So.. hopefully this short experience has sold you all already.
This script is currently a manual job and needs a bit of tweaking before we can put it in a cron job/run it always - it needs to watch for new commits and then run a modification of the above script on a per case basis (if two versions of it run in parallel we have an issue).
Definitely something we should push for next sprint!
Long live Barry bot!
Devs... (everyone else now of what follows is likely to be useful): I got the labs instance up and running on: http://gather-browser-tests.wmflabs.org/wiki/Main_Page
Most of you in readership team should be able to ssh gather-browser-tests.eqiad.wmflabs Let me know if you have no access.
It's currently working via a script that you can find here: /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh
[0] https://phabricator.wikimedia.org/T100293 [1]
https://gerrit.wikimedia.org/r/#/q/reviewer:jdlrobson%252Bbarry%2540gmail.co... [2] https://gerrit.wikimedia.org/r/#/c/218731/
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Jon Robson
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Dan Duvall Automation Engineer Wikimedia Foundation
-- Jon Robson
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Awesome stuff.
On Wed, Jun 24, 2015 at 8:25 PM, Jon Robson jdlrobson@gmail.com wrote:
I haven't been able to use verified yet - I think this is a permission that Barry doesn't have. I'm experimenting with him doing no score reviews right now rather than +1s for good patches. The -1s are worth being as visible as possible.
I'm hoping to send an update on the status quo but I've updated him so that he can be run in a script:
https://github.com/jdlrobson/Barry-the-Browser-Test-Bot/blob/master/barrybot... If you want him to re-review code simply remove him from the list of reviewers - the bot currently works out what it needs to review based on which patches he is currently not a reviewer for.
On Wed, Jun 24, 2015 at 5:36 AM, florian.schmidt.welzow@t-online.de florian.schmidt.welzow@t-online.de wrote:
How does it affect jenkins if something/somebody sets verified to -1 for example? Would it block it from merging?
Any -1/-2 in the verified column blocks submitting a change (even, if
other
reviewers add a +1 in verified), so if Barry wouldn't be happy, you can't merge. A workaround is to remove BarryBot as a reviewer (which would
remove
the vote, too, and allows jenkins to submit the change).
Freundliche Grüße Florian Schmidt
-----Original-Nachricht-----
Betreff: Re: [WikimediaMobile] [Update] Browser tests per patch
Datum: Wed, 24 Jun 2015 14:07:23 +0200
Von: Joaquin Oltra Hernandez jhernandez@wikimedia.org
An: "florian.schmidt.welzow@t-online.de" florian.schmidt.welzow@t-online.de
Baha I agree with you, but instead of staying silent just commenting a "BarryBot is happy! Browser tests passed" without setting any +1 would be good to know they run.
I also agree with Florian, if we could set Verified +1 or -1 that could
be
interesting. How does it affect jenkins if something/somebody sets
verified
to -1 for example? Would it block it from merging?
On Wed, Jun 24, 2015 at 1:53 PM, florian.schmidt.welzow@t-online.de florian.schmidt.welzow@t-online.de wrote:
Or, maybe, it should be possible for the bot to set the verified flag, instead of the code review (it's not a code review, so a minus one is misleading, too).
Freundliche Grüße Florian Schmidt
-----Original-Nachricht-----
Betreff: Re: [WikimediaMobile] [Update] Browser tests per patch
Datum: Wed, 24 Jun 2015 13:47:39 +0200
Von: Bahodir Mansurov bmansurov@wikimedia.org
An: Jon Robson jdlrobson@gmail.com
May I suggest that Barry should not +1 patches. It gives a false impression that the code is alright even though the code may not be
covered
at all. Can we have it just -1 when there is a problem, and stay silent otherwise?
On Jun 18, 2015, at 8:55 PM, Jon Robson jdlrobson@gmail.com wrote:
So far I'm seeing some extremely positive results. The tests are running super fast (in phantomjs the smoke tests are taking less than 3 minutes...).
I've been manually running him whilst I do code review in parallel and he's already generated some interesting conversation on this patchset: https://gerrit.wikimedia.org/r/#/c/219249/
If you want to pair and get this to be less hacky I'm more than happy!
On Thu, Jun 18, 2015 at 5:41 PM, Dan Duvall dduvall@wikimedia.org
wrote:
Nice work, Jon.
I've opened a task for defining a JJB builder/template and getting something like this into CI sooner rather than later.[1] I think your setup proves that a set of well-groomed MW-Selenium integration tests can be stable enough for this purpose, and we can start with an even smaller subset of core tests for a pre-merge build. Of course this isn't something that we planned to do 'now' now—sometimes 'then' suddenly becomes 'now' so 'soon'—but we can start with an experiment on Gather or MobileFrontend tests, since their health has greatly improved, and see how it goes.
[1] https://phabricator.wikimedia.org/T103039
On Thu, Jun 18, 2015 at 11:03 AM, Jon Robson jdlrobson@gmail.com
wrote:
So the script that actually runs the browser test is not in a generic useful form but it is: https://gist.github.com/jdlrobson/32b607f8009e897ee80c
It uses the GerritCommandLine tool to do grabbing and reviewing https://github.com/jdlrobson/GerritCommandLine
Ideally if we can use labs-tools-gerrit-to-redis for identifying patches and then pulling them down we wouldn't need the GerritCommandLine tool since the code to submit a review is pretty trivial and captured in this function:
https://github.com/jdlrobson/GerritCommandLine/blob/master/gerrit.py#L277
I've also put this in the task https://phabricator.wikimedia.org/T101069#1379462
We'll probably want an instance per extension, to simplify having to worry about dependencies (we can just run a git update on all extensions after each checkout)
On Thu, Jun 18, 2015 at 4:32 AM, Joaquin Oltra Hernandez jhernandez@wikimedia.org wrote:
Awesome Jon! I'm so happy to finally see this developing :DD
Loving the : `Browserbot happy!`
I've noticed it can report either the name of the failing test or the full log. What do you think if we show that, and a url with the pasted log somewhere publicly to not put too much noise on the comments but still be able to see it? Something like https://phabricator.wikimedia.org/paste/...
+1 to where is the source. +1 to documenting how you've set it all up on wiki somewhere.
I also think we need a catchy phrase for the -1s!
Thanks for you work on this, we'll get more focused time for it soon.
On Thu, Jun 18, 2015 at 11:33 AM, Sam Smith samsmith@wikimedia.org wrote:
I agree with Florian everything that you've written should be in a public version control system.
Second, I'd ask that you document your experiences so far in getting this set up and how it works so that other members of the vertical can help to maintain it moving forward.
Third, great work!!1
<3
–Sam
On Thu, Jun 18, 2015 at 7:09 AM, florian.schmidt.welzow@t-online.de florian.schmidt.welzow@t-online.de wrote:
It's currently working via a script that you can find here: /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh
It would be great to have the script in a public version control system (e.g. github?), especially for people, e.g. volunteers, who can't ssh to gather-browser-tests.eqiad.wmflabs[1]
[1] all people, who're not members of https://wikitech.wikimedia.org/wiki/Nova_Resource:Mobile-smoketests
Best, Florian
-----Original-Nachricht----- Betreff: [WikimediaMobile] [Update] Browser tests per patch Datum: Thu, 18 Jun 2015 03:27:32 +0200 Von: Jon Robson jrobson@wikimedia.org An: "QA (software quality assurance) for Wikimedia projects." qa@lists.wikimedia.org, mobile-l mobile-l@lists.wikimedia.org
Background: mobile wants to gain more confidence in its browser tests by running a subset of browser tests on a case by case basis [0].
Good news: I've got a proof of concept running and Barry the browser test bot has given some legitimate helpful reviews to Gather [1].
Even better news: It's proving itself valuable already [2]. As you can see in the messages the bot has posted on [3] we have a couple of options on display option format for his reviews.
So.. hopefully this short experience has sold you all already.
This script is currently a manual job and needs a bit of tweaking before we can put it in a cron job/run it always - it needs to watch for new commits and then run a modification of the above script on a per case basis (if two versions of it run in parallel we have an issue).
Definitely something we should push for next sprint!
Long live Barry bot!
Devs... (everyone else now of what follows is likely to be useful): I got the labs instance up and running on: http://gather-browser-tests.wmflabs.org/wiki/Main_Page
Most of you in readership team should be able to ssh gather-browser-tests.eqiad.wmflabs Let me know if you have no access.
It's currently working via a script that you can find here: /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh
https://gerrit.wikimedia.org/r/#/q/reviewer:jdlrobson%252Bbarry%2540gmail.co...
[2] https://gerrit.wikimedia.org/r/#/c/218731/
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Jon Robson
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Dan Duvall Automation Engineer Wikimedia Foundation
-- Jon Robson
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Jon Robson