I talked to Dan Duvall today and he said that many mobile browser tests fail on default vagrant instance because other extensions that are not hard dependencies of MF are not activated by default when activating the mobilefrontend role (extensions such as Echo, GeoData, VisualEditor). There are two things we can do:
1. Make other extensions that mobile uses dependencies of mobilefrontend role in vagrant 2. Create a separate role, e.g. mobilefrontend-browsertests, that will list mobilefrontend and all those extensions as its dependencies
I was wondering if there is anyone who uses vagrant and would like to have MF enabled, but not all the other extensions. If not, 1. seems like a simpler solution.
Thoughts?
I personally would prefer 2, because buncing everything into one role would be too inflexible. I don't see much maintenance overhead if compared with 1.
On Fri, Jul 18, 2014 at 6:19 PM, Juliusz Gonera jgonera@wikimedia.org wrote:
I talked to Dan Duvall today and he said that many mobile browser tests fail on default vagrant instance because other extensions that are not hard dependencies of MF are not activated by default when activating the mobilefrontend role (extensions such as Echo, GeoData, VisualEditor). There are two things we can do:
- Make other extensions that mobile uses dependencies of
mobilefrontend role in vagrant 2. Create a separate role, e.g. mobilefrontend-browsertests, that will list mobilefrontend and all those extensions as its dependencies
I was wondering if there is anyone who uses vagrant and would like to have MF enabled, but not all the other extensions. If not, 1. seems like a simpler solution.
Thoughts?
-- Juliusz
I agree with Max - or alternatively (or in addition) setting up a mobilefrontend-production role to set things up as close as possible to prod.
On Fri, Jul 18, 2014 at 9:22 PM, Max Semenik msemenik@wikimedia.org wrote:
I personally would prefer 2, because buncing everything into one role would be too inflexible. I don't see much maintenance overhead if compared with 1.
On Fri, Jul 18, 2014 at 6:19 PM, Juliusz Gonera jgonera@wikimedia.org wrote:
I talked to Dan Duvall today and he said that many mobile browser tests fail on default vagrant instance because other extensions that are not hard dependencies of MF are not activated by default when activating the mobilefrontend role (extensions such as Echo, GeoData, VisualEditor). There are two things we can do:
- Make other extensions that mobile uses dependencies of
mobilefrontend role in vagrant 2. Create a separate role, e.g. mobilefrontend-browsertests, that will list mobilefrontend and all those extensions as its dependencies
I was wondering if there is anyone who uses vagrant and would like to have MF enabled, but not all the other extensions. If not, 1. seems like a simpler solution.
Thoughts?
-- Juliusz
There is another alternative that might be worth getting opinions on—Juliusz and I both thought it seemed a bit complicated, but it would have some additional benefits, so here goes.
Firstly, we'd add cucumber tags to indicate the additional role dependencies of a given feature (or individual scenario). For instance, editor_ve.feature would include a `@visualeditor` tag and notification.feature would include an `@echo` tag. Secondly, we'd restrict the default behavior of cucumber (via puppet) to run only those tests that are tagged with any of the currently enabled roles.
The major upside to this approach would be that only those tests suited to the current development environment get run—ostensibly these are the only tests the developer would care to run—translating to far fewer false negatives and more confidence (and utility) in the test suite. With the dual-roles approach, it's highly likely that developers will continue to run into these soft-dependency issues. And, as already stated, a single-role approach is less flexible and will bloat the provision and git-update times.
The biggest downside to this approach is pretty obvious, I think: It would require the author of the test to know and include any role dependencies in the list of tags. Is this a reasonable expectation of developers and/or project managers (who might one day be writing features)? Another possible downside is that altering cucumber's default behavior might confuse people as to why some tests are being skipped.
On Tue, Jul 22, 2014 at 10:56 AM, Arthur Richards arichards@wikimedia.org wrote:
I agree with Max - or alternatively (or in addition) setting up a mobilefrontend-production role to set things up as close as possible to prod.
On Fri, Jul 18, 2014 at 9:22 PM, Max Semenik msemenik@wikimedia.org wrote:
I personally would prefer 2, because buncing everything into one role would be too inflexible. I don't see much maintenance overhead if compared with 1.
On Fri, Jul 18, 2014 at 6:19 PM, Juliusz Gonera jgonera@wikimedia.org wrote:
I talked to Dan Duvall today and he said that many mobile browser tests fail on default vagrant instance because other extensions that are not hard dependencies of MF are not activated by default when activating the mobilefrontend role (extensions such as Echo, GeoData, VisualEditor). There are two things we can do:
- Make other extensions that mobile uses dependencies of
mobilefrontend role in vagrant 2. Create a separate role, e.g. mobilefrontend-browsertests, that will list mobilefrontend and all those extensions as its dependencies
I was wondering if there is anyone who uses vagrant and would like to have MF enabled, but not all the other extensions. If not, 1. seems like a simpler solution.
Thoughts?
-- Juliusz
-- Arthur Richards Team Practices Lead [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687
Dan, thanks for elaborating on this third option.
I think it might be a good idea. I think the author of the test should know what the dependencies are. A few concerns:
* I guess it's a very early stage proposal, but I'd come up with a tag naming convention that clearly indicates what those tags do, like @role-echo (which marries the whole thing to Vagrant and its nomenclature) or @extension-echo.
* How would puppet alter default cucumber behavior?
* If we don't want to confuse people why some tests don't run, maybe we could store the list of available tags, check for available roles before running cucumber and show a message saying that tests requiring following (disabled) roles, will not run?
Chris, Zeljko, and I actually discussed some of these finer points earlier today. See my answers inline.
* I guess it's a very early stage proposal, but I'd come up with a tag
naming convention that clearly indicates what those tags do, like @role-echo (which marries the whole thing to Vagrant and its nomenclature) or @extension-echo.
A clearer naming convention is definitely a good idea. Not only will it avoid name collisions but I think it will promote a greater inference of what the tags mean—people will be less likely to copy, paste, ignore. Consider this might have applications beyond the scope of mw-vagrant, I'd prefer "@extension-" over "@role-".
- How would puppet alter default cucumber behavior?
We can configure puppet to write a default cucumber config https://github.com/cucumber/cucumber/wiki/cucumber.yml in the working directory for each test suite. The config will specify a default command-line invocation with `--tags` options to limit the scope of the run. It's possible that we'll need some sort of "@core" tag for features that don't have any extension dependency, but there may be another way—I'll have to experiment a bit.
- If we don't want to confuse people why some tests don't run, maybe
we could store the list of available tags, check for available roles before running cucumber and show a message saying that tests requiring following (disabled) roles, will not run?
Zeljko brought this up as well. We can probably implement some sort of after or exit hook to inform the user why some tests were skipped—based on the invocation maybe—and output it in yellow or some other alarming but not shit-your-pants alarming color. :)
-- Juliusz
On Tue, Jul 29, 2014 at 5:17 PM, Dan Duvall dduvall@wikimedia.org wrote:
There is another alternative that might be worth getting opinions
on—Juliusz
and I both thought it seemed a bit complicated, but it would have some additional benefits, so here goes.
Firstly, we'd add cucumber tags to indicate the additional role
dependencies
of a given feature (or individual scenario). For instance,
editor_ve.feature
would include a `@visualeditor` tag and notification.feature would
include
an `@echo` tag. Secondly, we'd restrict the default behavior of cucumber (via puppet) to run only those tests that are tagged with any of the currently enabled roles.
The major upside to this approach would be that only those tests suited
to
the current development environment get run—ostensibly these are the only tests the developer would care to run—translating to far fewer false negatives and more confidence (and utility) in the test suite. With the dual-roles approach, it's highly likely that developers will continue to
run
into these soft-dependency issues. And, as already stated, a single-role approach is less flexible and will bloat the provision and git-update
times.
The biggest downside to this approach is pretty obvious, I think: It
would
require the author of the test to know and include any role dependencies
in
the list of tags. Is this a reasonable expectation of developers and/or project managers (who might one day be writing features)? Another
possible
downside is that altering cucumber's default behavior might confuse
people
as to why some tests are being skipped.
On Tue, Jul 22, 2014 at 10:56 AM, Arthur Richards <
arichards@wikimedia.org>
wrote:
I agree with Max - or alternatively (or in addition) setting up a mobilefrontend-production role to set things up as close as possible to prod.
On Fri, Jul 18, 2014 at 9:22 PM, Max Semenik msemenik@wikimedia.org wrote:
I personally would prefer 2, because buncing everything into one role would be too inflexible. I don't see much maintenance overhead if
compared
with 1.
On Fri, Jul 18, 2014 at 6:19 PM, Juliusz Gonera <jgonera@wikimedia.org
wrote:
I talked to Dan Duvall today and he said that many mobile browser tests fail on default vagrant instance because other extensions that are not hard dependencies of MF are not activated by default when activating the mobilefrontend role (extensions such as Echo, GeoData, VisualEditor). There are two things we can do:
- Make other extensions that mobile uses dependencies of
mobilefrontend role in vagrant 2. Create a separate role, e.g. mobilefrontend-browsertests, that will list mobilefrontend and all those extensions as its dependencies
I was wondering if there is anyone who uses vagrant and would like to have MF enabled, but not all the other extensions. If not, 1. seems like a simpler solution.
Thoughts?
-- Juliusz
-- Arthur Richards Team Practices Lead [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687
-- Dan Duvall Automation Engineer Wikimedia Foundation
On Fri, Aug 1, 2014 at 1:50 AM, Dan Duvall dduvall@wikimedia.org wrote:
- If we don't want to confuse people why some tests don't run, maybe
we could store the list of available tags, check for available roles before running cucumber and show a message saying that tests requiring following (disabled) roles, will not run?
Zeljko brought this up as well. We can probably implement some sort of after or exit hook to inform the user why some tests were skipped—based on the invocation maybe—and output it in yellow or some other alarming but not shit-your-pants alarming color. :)
+1 :)
Željko
Some bad news: After some experimenting last week with the mw-vagrant/puppet side of things, it quickly became clear that writing out an rspec profile with the right tags would be difficult—trying to reflect on which roles are enabled in the puppet manifest is sort of a chicken-egg problem due to the declarative nature of the puppet language.
Good news: I was able to use the MediaWiki API from within mediawiki_selenium to implement a much more environment agnostic and generally less cumbersome system. So really, the bad news isn't all that bad, just moot.
https://gerrit.wikimedia.org/r/#/c/151802/
Zeljko or Chris (or Juliusz?),
The patch set depends on some API-client features that haven't yet been merged, namely support for siteinfo queries and response parsing. If you get a chance, I'd love to get the review moved along so we can cut a new gem version. I've implemented (currently passing) specs for the feature, so it seems good to go, but I wanted to get some additional eyes on it before doing a new release.
https://gerrit.wikimedia.org/r/#/c/150977/
Thanks!
On Mon, Aug 4, 2014 at 6:06 AM, Željko Filipin zfilipin@wikimedia.org wrote:
On Fri, Aug 1, 2014 at 1:50 AM, Dan Duvall dduvall@wikimedia.org wrote:
- If we don't want to confuse people why some tests don't run, maybe
we could store the list of available tags, check for available roles before running cucumber and show a message saying that tests requiring following (disabled) roles, will not run?
Zeljko brought this up as well. We can probably implement some sort of after or exit hook to inform the user why some tests were skipped—based on the invocation maybe—and output it in yellow or some other alarming but not shit-your-pants alarming color. :)
+1 :)
Željko
Example output: https://wikitech.wikimedia.org/wiki/File:Screen_Shot_2014-08-04_at_5.57.13_P...
Does the warning seem clear and prominent enough?
On Mon, Aug 4, 2014 at 5:38 PM, Dan Duvall dduvall@wikimedia.org wrote:
Some bad news: After some experimenting last week with the mw-vagrant/puppet side of things, it quickly became clear that writing out an rspec profile with the right tags would be difficult—trying to reflect on which roles are enabled in the puppet manifest is sort of a chicken-egg problem due to the declarative nature of the puppet language.
Good news: I was able to use the MediaWiki API from within mediawiki_selenium to implement a much more environment agnostic and generally less cumbersome system. So really, the bad news isn't all that bad, just moot.
https://gerrit.wikimedia.org/r/#/c/151802/
Zeljko or Chris (or Juliusz?),
The patch set depends on some API-client features that haven't yet been merged, namely support for siteinfo queries and response parsing. If you get a chance, I'd love to get the review moved along so we can cut a new gem version. I've implemented (currently passing) specs for the feature, so it seems good to go, but I wanted to get some additional eyes on it before doing a new release.
https://gerrit.wikimedia.org/r/#/c/150977/
Thanks!
On Mon, Aug 4, 2014 at 6:06 AM, Željko Filipin zfilipin@wikimedia.org wrote:
On Fri, Aug 1, 2014 at 1:50 AM, Dan Duvall dduvall@wikimedia.org wrote:
- If we don't want to confuse people why some tests don't run, maybe
we could store the list of available tags, check for available roles before running cucumber and show a message saying that tests requiring following (disabled) roles, will not run?
Zeljko brought this up as well. We can probably implement some sort of after or exit hook to inform the user why some tests were skipped—based on the invocation maybe—and output it in yellow or some other alarming but not shit-your-pants alarming color. :)
+1 :)
Željko
-- Dan Duvall Automation Engineer Wikimedia Foundation http://wikimediafoundation.org
On Tue, Aug 5, 2014 at 3:09 AM, Dan Duvall dduvall@wikimedia.org wrote:
Example output:
https://wikitech.wikimedia.org/wiki/File:Screen_Shot_2014-08-04_at_5.57.13_P... Does the warning seem clear and prominent enough?
Would adding a list of skipped features/scenarios be too much?
Željko
On Tue, Aug 5, 2014 at 2:38 AM, Dan Duvall dduvall@wikimedia.org wrote:
If you get a chance, I'd love to get the review moved along so we can cut a new gem version. I've implemented (currently passing) specs for the feature, so it seems good to go, but I wanted to get some additional eyes on it before doing a new release.
Let me know your rubygems.org username. I can add you to the owners of the gems, so you could publish new versions.
Željko
On Tue, Aug 5, 2014 at 2:38 AM, Dan Duvall dduvall@wikimedia.org wrote:
https://gerrit.wikimedia.org/r/#/c/151802/ (...) https://gerrit.wikimedia.org/r/#/c/150977/
I have left a few comments.
Željko
Great! Thanks, Zeljko.
I'm marxarelli https://rubygems.org/profiles/marxarelli on rubygems.
Regarding the skipped features, is it better to show the warning inline or a list at the end? I thought the latter might be too much noise. What about adding a "n skipped due to missing extensions" in the stats at the end and leaving the details inline?
On Wed, Aug 6, 2014 at 9:33 AM, Željko Filipin zfilipin@wikimedia.org wrote:
On Tue, Aug 5, 2014 at 2:38 AM, Dan Duvall dduvall@wikimedia.org wrote:
https://gerrit.wikimedia.org/r/#/c/151802/ (...) https://gerrit.wikimedia.org/r/#/c/150977/
I have left a few comments.
Željko
On Wed, Aug 6, 2014 at 6:50 PM, Dan Duvall dduvall@wikimedia.org wrote:
I'm marxarelli https://rubygems.org/profiles/marxarelli on rubygems.
You are now proud owner of two more gems[1].
Regarding the skipped features, is it better to show the warning inline or a list at the end? I thought the latter might be too much noise. What about adding a "n skipped due to missing extensions" in the stats at the end and leaving the details inline?
Both options sound good to me.
Željko -- 1: https://en.wikipedia.org/wiki/Uncle_Ben#.22With_great_power_comes_great_resp...