Hi everyone,
Some context here: Jinesh and the folks at Calcey are going to be pitching in on building out our Selenium test framework. I asked them to take a look here: http://svn.wikimedia.org/viewvc/mediawiki/trunk/phase3/maintenance/tests/sel...
...and then to suggest a plan of attack for building that out into a proper smoke test suite, as well as creating a larger battery of tests that we can run for comprehensive testing.
If people have thoughts on the types of things that developers inadvertently break that something like Selenium would be appropriate for trying to detect, your thoughts would be greatly appreciated here.
Jinesh, this looks like a great start. Comments inline:
On Thu, Nov 4, 2010 at 4:34 AM, Jinesh De Silva jinesh@calcey.com wrote:
More Test Cases could be add to the existing Selenium Test framework. The idea should be to test the main Wiki Editor functionalities and Page view/preview functionalities in a smoke test suite.
Yup, I agree, though one point we should probably clarify. The core smoke test should just test whether the plain editor works (without the WikiEditor extension). The WikiEditor extension should probably be tested as part of a suite of tests associated with all of the extensions we run in production.
By the way, here's a list of the extensions we run on English Wikipedia: http://en.wikipedia.org/wiki/Special:Version
...as well as the full list of extensions we currently have available in production: http://svn.wikimedia.org/viewvc/mediawiki/branches/wmf/1.16wmf4/extensions/
I don't have a strong opinion about which of these we should try tackling first, but generally testing the extensions that are most heavily used seems like a good thing to me.
The detailed administrative functionalities as User Preferences (more advanced features as Managing Watch-list, Advanced options in Editing, E-mail options) should be covered in a separate detailed test suite if necessary.
Also agreed. Perhaps others on this list can let us know: which aspects of these features are the most fragile + impact a lot of people when they break? My instinct is not to worry too much about these for now.
Wiki Editor functionalities can be tested by automating "Content Addition" in a new wiki page. All the basic text editing can be performed and should be verified in page preview and saved page view.
The same page can be tested for "Content Editing" by performing same editing functionalities in the previously saved page. The content changes can be verified in page preview and saved page view.
Set Basic preferences can be tested separately and basic settings as Basic
Information, Signature, Date Format, Skin can be tested. The applied preferences can be verified against the saved page.
The other utility functionalities as Search for Pages, Move Page, My Watch-list, My Contributions, Delete Pages can be tested at the end of the test suite. Delete Page functionality can be used to delete all the newly created pages.
So the smoke test suite can be of 4 main test suites;
Add content to new Page (Wiki Editor for new page) Edit content (Edit content in existing page) Basic user preferences (Basic Information, Signature, DateFormat, Skin)
Extra features (search, move, delete, watch-list,contribution)
All the above can be supported by some common functionalities as;
User Login Search for Pages Search and copy Text Select applied skin Get Page Name and other attributes (Page Headings/Titlesetc.)
Get text atttributes (bold, italic, font size etc.)
This looks like a really good plan of attack. Anyone else care to weigh in?
Rob