Dan, thanks for elaborating on this third option.
I think it might be a good idea. I think the author of the test should
know what the dependencies are. A few concerns:
* I guess it's a very early stage proposal, but I'd come up with a tag
naming convention that clearly indicates what those tags do, like
@role-echo (which marries the whole thing to Vagrant and its
nomenclature) or @extension-echo.
* How would puppet alter default cucumber behavior?
* If we don't want to confuse people why some tests don't run, maybe
we could store the list of available tags, check for available roles
before running cucumber and show a message saying that tests requiring
following (disabled) roles, will not run?
--
Juliusz
On Tue, Jul 29, 2014 at 5:17 PM, Dan Duvall <dduvall(a)wikimedia.org> wrote:
There is another alternative that might be worth
getting opinions on—Juliusz
and I both thought it seemed a bit complicated, but it would have some
additional benefits, so here goes.
Firstly, we'd add cucumber tags to indicate the additional role dependencies
of a given feature (or individual scenario). For instance, editor_ve.feature
would include a `@visualeditor` tag and notification.feature would include
an `@echo` tag. Secondly, we'd restrict the default behavior of cucumber
(via puppet) to run only those tests that are tagged with any of the
currently enabled roles.
The major upside to this approach would be that only those tests suited to
the current development environment get run—ostensibly these are the only
tests the developer would care to run—translating to far fewer false
negatives and more confidence (and utility) in the test suite. With the
dual-roles approach, it's highly likely that developers will continue to run
into these soft-dependency issues. And, as already stated, a single-role
approach is less flexible and will bloat the provision and git-update times.
The biggest downside to this approach is pretty obvious, I think: It would
require the author of the test to know and include any role dependencies in
the list of tags. Is this a reasonable expectation of developers and/or
project managers (who might one day be writing features)? Another possible
downside is that altering cucumber's default behavior might confuse people
as to why some tests are being skipped.
On Tue, Jul 22, 2014 at 10:56 AM, Arthur Richards <arichards(a)wikimedia.org>
wrote:
I agree with Max - or alternatively (or in addition) setting up a
mobilefrontend-production role to set things up as close as possible to
prod.
On Fri, Jul 18, 2014 at 9:22 PM, Max Semenik <msemenik(a)wikimedia.org>
wrote:
I personally would prefer 2, because buncing everything into one role
would be too inflexible. I don't see much maintenance overhead if compared
with 1.
On Fri, Jul 18, 2014 at 6:19 PM, Juliusz Gonera <jgonera(a)wikimedia.org>
wrote:
I talked to Dan Duvall today and he said that many mobile browser
tests fail on default vagrant instance because other extensions that
are not hard dependencies of MF are not activated by default when
activating the mobilefrontend role (extensions such as Echo, GeoData,
VisualEditor). There are two things we can do:
1. Make other extensions that mobile uses dependencies of
mobilefrontend role in vagrant
2. Create a separate role, e.g. mobilefrontend-browsertests, that will
list mobilefrontend and all those extensions as its dependencies
I was wondering if there is anyone who uses vagrant and would like to
have MF enabled, but not all the other extensions. If not, 1. seems
like a simpler solution.
Thoughts?
--
Juliusz
--
Arthur Richards
Team Practices Lead
[[User:Awjrichards]]
IRC: awjr
+1-415-839-6885 x6687
--
Dan Duvall
Automation Engineer
Wikimedia Foundation