Is there a way to have the general mobile web automation tests run continuously, alerting #wikimedia-mobile and select individuals in the case there are failing tests?
Sorry if this has been asked and answered, or is already the case and I missed it somehow.
A bug introduced to ZeroRatedMobileAccess interfered with MobileFrontend yesterday. It was spotted manually. It would be cool to increase the odds of automatically being notified of failing tests from the general mobile web (the Wikipedia Zero automation tests do positive/negative tests for in-scope Wikipedia Zero, and only a couple of negative tests for the general mobile web). To be fair, tests won't always catch errant behavior, but they improve the odds!
Props to Arthur, Jon, Juliusz, and Max spotting the issue last night and Yuri and Yuvi (Yuri !== Yuvi) getting the fix in shortly thereafter ( https://bugzilla.wikimedia.org/show_bug.cgi?id=54209).
Thanks. -Adam
Sure can are these tests already setup in cloudbees to run?
On Tue, Sep 17, 2013 at 11:17 AM, Adam Baso abaso@wikimedia.org wrote:
Is there a way to have the general mobile web automation tests run continuously, alerting #wikimedia-mobile and select individuals in the case there are failing tests?
Sorry if this has been asked and answered, or is already the case and I missed it somehow.
A bug introduced to ZeroRatedMobileAccess interfered with MobileFrontend yesterday. It was spotted manually. It would be cool to increase the odds of automatically being notified of failing tests from the general mobile web (the Wikipedia Zero automation tests do positive/negative tests for in-scope Wikipedia Zero, and only a couple of negative tests for the general mobile web). To be fair, tests won't always catch errant behavior, but they improve the odds!
Props to Arthur, Jon, Juliusz, and Max spotting the issue last night and Yuri and Yuvi (Yuri !== Yuvi) getting the fix in shortly thereafter ( https://bugzilla.wikimedia.org/show_bug.cgi?id=54209).
Thanks. -Adam
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
To follow up with the lists, it seems that there are already continuously running tests that email mobile-l upon failure, but we were unlucky (or lucky, depending on your worldview) that failures weren't triggered because the core article reading experience in the mobile web in general continued to hum along.
Consequently, we're working with Michelle on defining a few positive and negative tests relating to image tags, banners, and external hyperlinks.
Thanks all! -Adam
On Tue, Sep 17, 2013 at 11:32 AM, Michelle Grover mgrover@wikimedia.orgwrote:
Sure can are these tests already setup in cloudbees to run?
On Tue, Sep 17, 2013 at 11:17 AM, Adam Baso abaso@wikimedia.org wrote:
Is there a way to have the general mobile web automation tests run continuously, alerting #wikimedia-mobile and select individuals in the case there are failing tests?
Sorry if this has been asked and answered, or is already the case and I missed it somehow.
A bug introduced to ZeroRatedMobileAccess interfered with MobileFrontend yesterday. It was spotted manually. It would be cool to increase the odds of automatically being notified of failing tests from the general mobile web (the Wikipedia Zero automation tests do positive/negative tests for in-scope Wikipedia Zero, and only a couple of negative tests for the general mobile web). To be fair, tests won't always catch errant behavior, but they improve the odds!
Props to Arthur, Jon, Juliusz, and Max spotting the issue last night and Yuri and Yuvi (Yuri !== Yuvi) getting the fix in shortly thereafter ( https://bugzilla.wikimedia.org/show_bug.cgi?id=54209).
Thanks. -Adam
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Let us know if we can help with that... -Chris
On Tue, Sep 17, 2013 at 12:09 PM, Adam Baso abaso@wikimedia.org wrote:
To follow up with the lists, it seems that there are already continuously running tests that email mobile-l upon failure, but we were unlucky (or lucky, depending on your worldview) that failures weren't triggered because the core article reading experience in the mobile web in general continued to hum along.
Consequently, we're working with Michelle on defining a few positive and negative tests relating to image tags, banners, and external hyperlinks.
Thanks all! -Adam
On Tue, Sep 17, 2013 at 11:32 AM, Michelle Grover mgrover@wikimedia.orgwrote:
Sure can are these tests already setup in cloudbees to run?
On Tue, Sep 17, 2013 at 11:17 AM, Adam Baso abaso@wikimedia.org wrote:
Is there a way to have the general mobile web automation tests run continuously, alerting #wikimedia-mobile and select individuals in the case there are failing tests?
Sorry if this has been asked and answered, or is already the case and I missed it somehow.
A bug introduced to ZeroRatedMobileAccess interfered with MobileFrontend yesterday. It was spotted manually. It would be cool to increase the odds of automatically being notified of failing tests from the general mobile web (the Wikipedia Zero automation tests do positive/negative tests for in-scope Wikipedia Zero, and only a couple of negative tests for the general mobile web). To be fair, tests won't always catch errant behavior, but they improve the odds!
Props to Arthur, Jon, Juliusz, and Max spotting the issue last night and Yuri and Yuvi (Yuri !== Yuvi) getting the fix in shortly thereafter ( https://bugzilla.wikimedia.org/show_bug.cgi?id=54209).
Thanks. -Adam
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
QA mailing list QA@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/qa
On Tue, Sep 17, 2013 at 8:17 PM, Adam Baso abaso@wikimedia.org wrote:
Is there a way to have the general mobile web automation tests run continuously, alerting #wikimedia-mobile and select individuals in the case there are failing tests?
As you have noticed, we are already doing that at Cloudbees[1]. (Jenkins jobs should soon be moved to integration.wikimedia.org[2]). All you need to do now is to "just" write some tests. :)
Let us know if you need help with that.
Željko -- 1: https://wmf.ci.cloudbees.com/view/r-mf/ 2: https://integration.wikimedia.org/ci/job/qa-browsertests-run/
Thanks! I believe Michelle is writing a couple of simple tests for MobileFrontend that, depending on the behavior, would connote badly behaving ZeroRatedMobileAccess or, perhaps, something else broken.
I have some PhantomJS stuff I will need to figure out how to integrate. Will be in touch later on that.
-Adam
On Tue, Sep 24, 2013 at 4:29 AM, Željko Filipin zfilipin@wikimedia.orgwrote:
On Tue, Sep 17, 2013 at 8:17 PM, Adam Baso abaso@wikimedia.org wrote:
Is there a way to have the general mobile web automation tests run continuously, alerting #wikimedia-mobile and select individuals in the case there are failing tests?
As you have noticed, we are already doing that at Cloudbees[1]. (Jenkins jobs should soon be moved to integration.wikimedia.org[2]). All you need to do now is to "just" write some tests. :)
Let us know if you need help with that.
Željko
1: https://wmf.ci.cloudbees.com/view/r-mf/ 2: https://integration.wikimedia.org/ci/job/qa-browsertests-run/
QA mailing list QA@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/qa
Right I planned to do those in the cloudbees instance where the rest of the MobileFrontend tests are and just have them as part of our tests.
On Tue, Sep 24, 2013 at 2:52 PM, Adam Baso abaso@wikimedia.org wrote:
Thanks! I believe Michelle is writing a couple of simple tests for MobileFrontend that, depending on the behavior, would connote badly behaving ZeroRatedMobileAccess or, perhaps, something else broken.
I have some PhantomJS stuff I will need to figure out how to integrate. Will be in touch later on that.
-Adam
On Tue, Sep 24, 2013 at 4:29 AM, Željko Filipin zfilipin@wikimedia.orgwrote:
On Tue, Sep 17, 2013 at 8:17 PM, Adam Baso abaso@wikimedia.org wrote:
Is there a way to have the general mobile web automation tests run continuously, alerting #wikimedia-mobile and select individuals in the case there are failing tests?
As you have noticed, we are already doing that at Cloudbees[1]. (Jenkins jobs should soon be moved to integration.wikimedia.org[2]). All you need to do now is to "just" write some tests. :)
Let us know if you need help with that.
Željko
1: https://wmf.ci.cloudbees.com/view/r-mf/ 2: https://integration.wikimedia.org/ci/job/qa-browsertests-run/
QA mailing list QA@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/qa
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l