I just +2'ed a change to add a few basic selenium tests to core [1]. I think it will benefit us all to have a set of automated tests to quickly make sure mediawiki is working correctly. From a security perspective, this also takes a step towards more efficient security testing, which I'm also a fan of (if you've tried blindly scanning mediawiki, you know what I'm talking about..).
I think the QA group is working on vagrant-izing these, but if you have ruby >1.9.3 and firefox, then setting up and running these tests on your local dev system is 4-6 commands,
$ cd tests/browser $ gem update --system $ gem install bundler $ bundle install
You can either set your environment variables yourself, or edit environment_variables and run `source environment_variables` to set them. Then it's just
$ bundle exec cucumber features/
to run the tests. They currently complete in 36 seconds on my laptop.
I'd like to see more tests added and backported to REL1_23 to make sure we have an ongoing suite to check releases against for next few years that we support that LTS. If anyone is interested in both mediawiki core and browser tests, I'm sure the QA team would like to get you involved.
Big thanks to hashar, Chris McMahon, and Dan Duvall for indulging me and getting this done. I'll let them jump in with all the details I've missed.
On 24 June 2014 15:46, Chris Steipp csteipp@wikimedia.org wrote:
I just +2'ed a change to add a few basic selenium tests to core [1]. I think it will benefit us all to have a set of automated tests to quickly make sure mediawiki is working correctly.
Great news; thank you all!
J.
Nice work guys! One slight issue though. The user Selenium_user (as set in environment_variables) was just indefinitely blocked on en.wiki :(
Ryan Kaldari
On Tue, Jun 24, 2014 at 3:46 PM, Chris Steipp csteipp@wikimedia.org wrote:
I just +2'ed a change to add a few basic selenium tests to core [1]. I think it will benefit us all to have a set of automated tests to quickly make sure mediawiki is working correctly. From a security perspective, this also takes a step towards more efficient security testing, which I'm also a fan of (if you've tried blindly scanning mediawiki, you know what I'm talking about..).
I think the QA group is working on vagrant-izing these, but if you have ruby >1.9.3 and firefox, then setting up and running these tests on your local dev system is 4-6 commands,
$ cd tests/browser $ gem update --system $ gem install bundler $ bundle install
You can either set your environment variables yourself, or edit environment_variables and run `source environment_variables` to set them. Then it's just
$ bundle exec cucumber features/
to run the tests. They currently complete in 36 seconds on my laptop.
I'd like to see more tests added and backported to REL1_23 to make sure we have an ongoing suite to check releases against for next few years that we support that LTS. If anyone is interested in both mediawiki core and browser tests, I'm sure the QA team would like to get you involved.
Big thanks to hashar, Chris McMahon, and Dan Duvall for indulging me and getting this done. I'll let them jump in with all the details I've missed.
[1] - https://gerrit.wikimedia.org/r/#/c/133507/
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
<quote name="Ryan Kaldari" date="2014-06-24" time="15:56:31 -0700">
Nice work guys! One slight issue though. The user Selenium_user (as set in environment_variables) was just indefinitely blocked on en.wiki :(
That shouldn't affect these tests. These run on your local box/vagrant.
On Tue, Jun 24, 2014 at 3:56 PM, Ryan Kaldari rkaldari@wikimedia.org wrote:
Nice work guys! One slight issue though. The user Selenium_user (as set in environment_variables) was just indefinitely blocked on en.wiki :(
Yes, and I think we should keep it that way. See my msg. to the mobile-l and qa lists earlier today: http://lists.wikimedia.org/pipermail/mobile-l/2014-June/007435.html
These tests are intended as acceptance tests for a new mediawiki install and smoke tests for a test env like beta labs. Let's not run them against production.
-Chris
Ryan Kaldari
On Tue, Jun 24, 2014 at 3:46 PM, Chris Steipp csteipp@wikimedia.org wrote:
I just +2'ed a change to add a few basic selenium tests to core [1]. I think it will benefit us all to have a set of automated tests to quickly make sure mediawiki is working correctly. From a security perspective, this also takes a step towards more efficient security testing, which I'm also a fan of (if you've tried blindly scanning mediawiki, you know what I'm talking about..).
I think the QA group is working on vagrant-izing these, but if you have ruby >1.9.3 and firefox, then setting up and running these tests on your local dev system is 4-6 commands,
$ cd tests/browser $ gem update --system $ gem install bundler $ bundle install
You can either set your environment variables yourself, or edit environment_variables and run `source environment_variables` to set them. Then it's just
$ bundle exec cucumber features/
to run the tests. They currently complete in 36 seconds on my laptop.
I'd like to see more tests added and backported to REL1_23 to make sure we have an ongoing suite to check releases against for next few years that we support that LTS. If anyone is interested in both mediawiki core and browser tests, I'm sure the QA team would like to get you involved.
Big thanks to hashar, Chris McMahon, and Dan Duvall for indulging me and getting this done. I'll let them jump in with all the details I've missed.
[1] - https://gerrit.wikimedia.org/r/#/c/133507/
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On 24 June 2014 18:56, Ryan Kaldari rkaldari@wikimedia.org wrote:
Nice work guys! One slight issue though. The user Selenium_user (as set in environment_variables) was just indefinitely blocked on en.wiki :(
Sorry to be a bit OT, but if you guys are going to test, please don't do it in article space on enwiki, or this is what is going to happen to the accounts. We've had to almost kick WMF staff off enwiki before because they kept testing in live article space, please don't do that.
Just be glad that the blocking admin didn't make it a block with IP autoblocked for 24 hours, or there would have been a bigger problem.
Testwiki is for test....and if you must test on enwiki, do it in userspace.
Risker/Anne
On Jun 24, 2014 9:05 PM, "Risker" risker.wp@gmail.com wrote:
On 24 June 2014 18:56, Ryan Kaldari rkaldari@wikimedia.org wrote:
Nice work guys! One slight issue though. The user Selenium_user (as set
in
environment_variables) was just indefinitely blocked on en.wiki :(
Sorry to be a bit OT, but if you guys are going to test, please don't do
it
in article space on enwiki, or this is what is going to happen to the accounts. We've had to almost kick WMF staff off enwiki before because they kept testing in live article space, please don't do that.
Just be glad that the blocking admin didn't make it a block with IP autoblocked for 24 hours, or there would have been a bigger problem.
Testwiki is for test....and if you must test on enwiki, do it in
userspace.
Risker/Anne
That sounds entirely reasonable to me.
--bawolff
On 24 June 2014 17:05, Risker risker.wp@gmail.com wrote:
Sorry to be a bit OT, but if you guys are going to test, please don't do it in article space on enwiki, or this is what is going to happen to the accounts. We've had to almost kick WMF staff off enwiki before because they kept testing in live article space, please don't do that.
Or you'll accidentally insert vandalism into articles when testing the abuse filter. Because I've *totally never* done that by accident. :-)
If these browsers tests don't interact with the site that leaves anything user-facing behind (e.g. making edits or take actions that create log entries), there's no problem with running them on enwiki. Otherwise, they should be using our test or beta sites.
Dan
On Jun 24, 2014 6:13 PM, "Dan Garry" dgarry@wikimedia.org wrote:
On 24 June 2014 17:05, Risker risker.wp@gmail.com wrote:
Sorry to be a bit OT, but if you guys are going to test, please don't
do it
in article space on enwiki, or this is what is going to happen to the accounts. We've had to almost kick WMF staff off enwiki before because they kept testing in live article space, please don't do that.
Or you'll accidentally insert vandalism into articles when testing the abuse filter. Because I've *totally never* done that by accident. :-)
If these browsers tests don't interact with the site that leaves anything user-facing behind (e.g. making edits or take actions that create log entries), there's no problem with running them on enwiki. Otherwise, they should be using our test or beta sites.
They do leave some artifacts (documented in the change, iirc). So yeah, they should only be used against your dev environment or beta.
Dan
-- Dan Garry Associate Product Manager for Platform and Mobile Apps Wikimedia Foundation _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Thanks for reviewing that, Chris!
Chris McMahon did most of the hard work there, porting everything from qa/browsertests into core. I was glad to help on the Vagrant side of things, though: I have a mediawiki-vagrant commit standing by that will do all the heavy heavy installation lifting that you mentioned. :) I'll double check that it's backwards compatible and send it up for review shortly.
On Tue, Jun 24, 2014 at 3:46 PM, Chris Steipp csteipp@wikimedia.org wrote:
I just +2'ed a change to add a few basic selenium tests to core [1]. I think it will benefit us all to have a set of automated tests to quickly make sure mediawiki is working correctly. From a security perspective, this also takes a step towards more efficient security testing, which I'm also a fan of (if you've tried blindly scanning mediawiki, you know what I'm talking about..).
I think the QA group is working on vagrant-izing these, but if you have ruby >1.9.3 and firefox, then setting up and running these tests on your local dev system is 4-6 commands,
$ cd tests/browser $ gem update --system $ gem install bundler $ bundle install
You can either set your environment variables yourself, or edit environment_variables and run `source environment_variables` to set them. Then it's just
$ bundle exec cucumber features/
to run the tests. They currently complete in 36 seconds on my laptop.
I'd like to see more tests added and backported to REL1_23 to make sure we have an ongoing suite to check releases against for next few years that we support that LTS. If anyone is interested in both mediawiki core and browser tests, I'm sure the QA team would like to get you involved.
Big thanks to hashar, Chris McMahon, and Dan Duvall for indulging me and getting this done. I'll let them jump in with all the details I've missed.
[1] - https://gerrit.wikimedia.org/r/#/c/133507/
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Le 25/06/2014 00:46, Chris Steipp a écrit :
I just +2'ed a change to add a few basic selenium tests to core [1]. I think it will benefit us all to have a set of automated tests to quickly make sure mediawiki is working correctly. From a security perspective, this also takes a step towards more efficient security testing, which I'm also a fan of (if you've tried blindly scanning mediawiki, you know what I'm talking about..).
<snip>
The next big step would be to trigger them whenever someone propose a patchset in Gerrit or vote +2. We have some scripts to setup a MediaWiki locally and run the browser tests, but that proven to be a bit unstable :-/
I am wondering how developer would feel with having browser tests reported back to Gerrit. Possibly has a different check so we dont have to wait for them to complete before reporting all the other tests.
I'd like to see more tests added and backported to REL1_23 to make sure we have an ongoing suite to check releases against for next few years that we support that LTS. If anyone is interested in both mediawiki core and browser tests, I'm sure the QA team would like to get you involved.
That is a nice idea. We could then trigger them before releasing a new tarball which will add some confidence. There is a bit of script needed there but that is definitely doable.
Big thanks to hashar, Chris McMahon, and Dan Duvall for indulging me and getting this done. I'll let them jump in with all the details I've missed.
Oh I have done nothing beside requesting someone to write the announcement :-]
wikitech-l@lists.wikimedia.org