Historically developers have had to setup their own content in mediawiki and in mediawiki-vagrant. While this can be done with a simple import, getting everyone on the same page is apparently not as easy. This is generally problematic as we would like to test code locally and remotely with the same content for various reasons.
Slightly more frustrating, there are pages titled "0.4425590476103759" on beta labs. While trying to sign off on a feature, there is usually a struggle when trying to find an article with suitable content. AFAIK this won't change beta labs but would provide a nice standard for our content on test wikis.
We aim to better things by creating a vagrant role for importing a set of articles for testing purposes. For more information please see related phabricator tasks [1] and [2].
In hopes of making this a nice collection of articles that multiple teams would use, we would like to get input from our designers and devs on what types of articles should be in this import. What qualities should these articles contain?
1: https://phabricator.wikimedia.org/T104561 2: https://phabricator.wikimedia.org/T62116
Given the topic, let's keep the QA list in the loop on this so the MW-Vagrant maintainers can participate/see.
Also, it looks like the original bug (reported in the MW-Vagrant project) covers this specific request from Reading, no? Essentially, let's see how far we can get with a general "MW-Vagrant (WMF?) testing data import" instead of a vertical specific "reading-web test data set". If what the Reading team needs is way too much for this then we can break it out, otherwise it seems like a needless distinction.
Greg
PS: https://wikitech.wikimedia.org/wiki/Labs_labs_labs
<quote name="Rob Moen" date="2015-07-20" time="17:11:07 -0700">
Historically developers have had to setup their own content in mediawiki and in mediawiki-vagrant. While this can be done with a simple import, getting everyone on the same page is apparently not as easy. This is generally problematic as we would like to test code locally and remotely with the same content for various reasons.
Slightly more frustrating, there are pages titled "0.4425590476103759" on beta labs. While trying to sign off on a feature, there is usually a struggle when trying to find an article with suitable content. AFAIK this won't change beta labs but would provide a nice standard for our content on test wikis.
We aim to better things by creating a vagrant role for importing a set of articles for testing purposes. For more information please see related phabricator tasks [1] and [2].
In hopes of making this a nice collection of articles that multiple teams would use, we would like to get input from our designers and devs on what types of articles should be in this import. What qualities should these articles contain?
1: https://phabricator.wikimedia.org/T104561 2: https://phabricator.wikimedia.org/T62116
-- Rob Moen Wikimedia Foundation rmoen@wikimedia.org
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
On 20 Jul 2015 5:56 pm, "Greg Grossmeier" greg@wikimedia.org wrote:
Given the topic, let's keep the QA list in the loop on this so the MW-Vagrant maintainers can participate/see.
Great :)
Also, it looks like the original bug (reported in the MW-Vagrant project) covers this specific request from Reading, no? Essentially,
let's see how far we can get with a general "MW-Vagrant (WMF?) testing
data import" instead of a vertical specific "reading-web test data set". If what the Reading team needs is way too much for this then we can break it out, otherwise it seems like a needless distinction.
It does yup. I've already tagged the bug with it. I'm hoping by tackling this we can come up with a common solution. The way I imagine this working in future is we have various vagrant roles for stock data e.g. reading-web-stock-data, editing-web-stock-data, sad-web-stock-data There would also be non team specific stock data that might be a sub role of this, for example, the reading web team commonly has to setup the wikidata role and manually create articles in the wikidata instance and local instance that are tied to each other - this takes a ridiculous amount of time and is one I'm keen to automate, given that we are leaning more heavily on wikidata descriptions and other data in there.
Rob - I've setup https://www.mediawiki.org/wiki/Reading/QA/Sample_articles as a place we can start to collect and think about these pages.
Greg
PS: https://wikitech.wikimedia.org/wiki/Labs_labs_labs
<quote name="Rob Moen" date="2015-07-20" time="17:11:07 -0700"> > Historically developers have had to setup their own content in mediawiki > and in mediawiki-vagrant. While this can be done with a simple import, > getting everyone on the same page is apparently not as easy. This is > generally problematic as we would like to test code locally and remotely > with the same content for various reasons. > > Slightly more frustrating, there are pages titled "0.4425590476103759" on > beta labs. While trying to sign off on a feature, there is usually a > struggle when trying to find an article with suitable content. AFAIK this > won't change beta labs but would provide a nice standard for our content on > test wikis. > > We aim to better things by creating a vagrant role for importing a set of > articles for testing purposes. For more information please see related > phabricator tasks [1] and [2]. > > In hopes of making this a nice collection of articles that multiple teams > would use, we would like to get input from our designers and devs on what > types of articles should be in this import. What qualities should these > articles contain? > > 1: https://phabricator.wikimedia.org/T104561 > 2: https://phabricator.wikimedia.org/T62116 > > > > -- > Rob Moen > Wikimedia Foundation > rmoen@wikimedia.org
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @greg A18D 1138 8E47 FAC8 1C7D |
QA mailing list QA@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/qa
I like this idea for development and manual testing, but I'm not sure it's appropriate for automated testing.
Generally in automated testing, each test case (unit test or integration scenario) should be able to set up or guarantee its preconditions whenever possible—and otherwise communicate its assumptions—which in the context of MediaWiki includes user accounts, user settings, or article content that the test case will manipulate or otherwise depend on. Putting this initial content in MW-Vagrant, which is orthogonal to MW test suites, creates the need for more coupling between test code and test environments and promotes more non-deterministic test behavior, something we've been trying very hard to reduce.[1][2]
Also, for development content, we might want to figure out a better place for it than in the MW-Vagrant repo itself. Article dumps are likely to build up fast, and we risk bloating the project repo itself with big blobs.
[1] https://www.mediawiki.org/wiki/Quality_Assurance/Browser_testing/Environment... [2] https://doc.wikimedia.org/rubygems/mediawiki-selenium/index.html#User_Factor...
On Tue, Jul 21, 2015 at 10:25 AM, Jon Robson jdlrobson@gmail.com wrote:
On 20 Jul 2015 5:56 pm, "Greg Grossmeier" greg@wikimedia.org wrote:
Given the topic, let's keep the QA list in the loop on this so the MW-Vagrant maintainers can participate/see.
Great :)
Also, it looks like the original bug (reported in the MW-Vagrant project) covers this specific request from Reading, no? Essentially,
let's see how far we can get with a general "MW-Vagrant (WMF?) testing
data import" instead of a vertical specific "reading-web test data set". If what the Reading team needs is way too much for this then we can break it out, otherwise it seems like a needless distinction.
It does yup. I've already tagged the bug with it. I'm hoping by tackling this we can come up with a common solution. The way I imagine this working in future is we have various vagrant roles for stock data e.g. reading-web-stock-data, editing-web-stock-data, sad-web-stock-data There would also be non team specific stock data that might be a sub role of this, for example, the reading web team commonly has to setup the wikidata role and manually create articles in the wikidata instance and local instance that are tied to each other - this takes a ridiculous amount of time and is one I'm keen to automate, given that we are leaning more heavily on wikidata descriptions and other data in there.
Rob - I've setup https://www.mediawiki.org/wiki/Reading/QA/Sample_articles as a place we can start to collect and think about these pages.
Greg
PS: https://wikitech.wikimedia.org/wiki/Labs_labs_labs
<quote name="Rob Moen" date="2015-07-20" time="17:11:07 -0700"> > Historically developers have had to setup their own content in mediawiki > and in mediawiki-vagrant. While this can be done with a simple import, > getting everyone on the same page is apparently not as easy. This is > generally problematic as we would like to test code locally and remotely > with the same content for various reasons. > > Slightly more frustrating, there are pages titled "0.4425590476103759" on > beta labs. While trying to sign off on a feature, there is usually a > struggle when trying to find an article with suitable content. AFAIK this > won't change beta labs but would provide a nice standard for our content on > test wikis. > > We aim to better things by creating a vagrant role for importing a set of > articles for testing purposes. For more information please see related > phabricator tasks [1] and [2]. > > In hopes of making this a nice collection of articles that multiple teams > would use, we would like to get input from our designers and devs on what > types of articles should be in this import. What qualities should these > articles contain? > > 1: https://phabricator.wikimedia.org/T104561 > 2: https://phabricator.wikimedia.org/T62116 > > > > -- > Rob Moen > Wikimedia Foundation > rmoen@wikimedia.org
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @greg A18D 1138 8E47 FAC8 1C7D |
QA mailing list QA@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/qa
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
iOS in particular would be _very_ interested in this, but would be more than happy to start testing against pages in beta labs or even local fixtures as a baby-step forward.
As far as test data being orthogonal to vagrant, they don't have to be coupled. I've had basic experience "pre-baking" elasticsearch docker containers with documents and verifying that a wrapper service can execute basic fuzzy queries against it. Good as a high-level sanity check. The "pre-baking" involved pulling the latest "vanilla" container (i.e. MW Vagrant in our case), injecting it with test documents, then committing and pushing the result. This approach could be applied here in a way that keeps MW vagrant and test data concerns and artifacts separate.
Also, one neat thing I saw done at Spotify was a "Test Data Service." This service's job was to abstract away concrete test data from its qualities. This was mostly used to acquire test users, thus allowing testers to declaratively acquire test data. For example, you could say "I want a free user with X flags" or "I want a premium user who has tracks in their starred playlist." The service's other job was to lock users to prevent parallel tests from corrupting a shared test user's state (also resulting in a false negative).
We might want something similar for retrieving and locking test articles for editing, but this might not be necessary if our vagrant/virtualization infrastructure is mature enough to ensure tests are isolated.
On Wed, Jul 22, 2015 at 12:58 PM, Dan Duvall dduvall@wikimedia.org wrote:
I like this idea for development and manual testing, but I'm not sure it's appropriate for automated testing.
Generally in automated testing, each test case (unit test or integration scenario) should be able to set up or guarantee its preconditions whenever possible—and otherwise communicate its assumptions—which in the context of MediaWiki includes user accounts, user settings, or article content that the test case will manipulate or otherwise depend on. Putting this initial content in MW-Vagrant, which is orthogonal to MW test suites, creates the need for more coupling between test code and test environments and promotes more non-deterministic test behavior, something we've been trying very hard to reduce.[1][2]
Also, for development content, we might want to figure out a better place for it than in the MW-Vagrant repo itself. Article dumps are likely to build up fast, and we risk bloating the project repo itself with big blobs.
[1] https://www.mediawiki.org/wiki/Quality_Assurance/Browser_testing/Environment... [2] https://doc.wikimedia.org/rubygems/mediawiki-selenium/index.html#User_Factor...
On Tue, Jul 21, 2015 at 10:25 AM, Jon Robson jdlrobson@gmail.com wrote:
On 20 Jul 2015 5:56 pm, "Greg Grossmeier" greg@wikimedia.org wrote:
Given the topic, let's keep the QA list in the loop on this so the MW-Vagrant maintainers can participate/see.
Great :)
Also, it looks like the original bug (reported in the MW-Vagrant project) covers this specific request from Reading, no? Essentially,
let's see how far we can get with a general "MW-Vagrant (WMF?) testing
data import" instead of a vertical specific "reading-web test data set". If what the Reading team needs is way too much for this then we can break it out, otherwise it seems like a needless distinction.
It does yup. I've already tagged the bug with it. I'm hoping by tackling this we can come up with a common solution. The way I imagine this working in future is we have various vagrant roles for stock data e.g. reading-web-stock-data, editing-web-stock-data, sad-web-stock-data There would also be non team specific stock data that might be a sub role of this, for example, the reading web team commonly has to setup the wikidata role and manually create articles in the wikidata instance and local instance that are tied to each other - this takes a ridiculous amount of time and is one I'm keen to automate, given that we are leaning more heavily on wikidata descriptions and other data in there.
Rob - I've setup https://www.mediawiki.org/wiki/Reading/QA/Sample_articles as a place we can start to collect and think about these pages.
Greg
PS: https://wikitech.wikimedia.org/wiki/Labs_labs_labs
<quote name="Rob Moen" date="2015-07-20" time="17:11:07 -0700"> > Historically developers have had to setup their own content in mediawiki > and in mediawiki-vagrant. While this can be done with a simple import, > getting everyone on the same page is apparently not as easy. This is > generally problematic as we would like to test code locally and remotely > with the same content for various reasons. > > Slightly more frustrating, there are pages titled "0.4425590476103759" on > beta labs. While trying to sign off on a feature, there is usually a > struggle when trying to find an article with suitable content. AFAIK this > won't change beta labs but would provide a nice standard for our content on > test wikis. > > We aim to better things by creating a vagrant role for importing a set of > articles for testing purposes. For more information please see related > phabricator tasks [1] and [2]. > > In hopes of making this a nice collection of articles that multiple teams > would use, we would like to get input from our designers and devs on what > types of articles should be in this import. What qualities should these > articles contain? > > 1: https://phabricator.wikimedia.org/T104561 > 2: https://phabricator.wikimedia.org/T62116 > > > > -- > Rob Moen > Wikimedia Foundation > rmoen@wikimedia.org
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @greg A18D 1138 8E47 FAC8 1C7D |
QA mailing list QA@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/qa
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Dan Duvall Automation Engineer Wikimedia Foundation http://wikimediafoundation.org
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
On Wed, Jul 22, 2015 at 10:43 AM, Brian Gerstle bgerstle@wikimedia.org wrote:
Also, one neat thing I saw done at Spotify was a "Test Data Service." This service's job was to abstract away concrete test data from its qualities. This was mostly used to acquire test users, thus allowing testers to declaratively acquire test data. For example, you could say "I want a free user with X flags" or "I want a premium user who has tracks in their starred playlist." The service's other job was to lock users to prevent parallel tests from corrupting a shared test user's state (also resulting in a false negative).
That's really interesting. Do you have a link to further reading/docs?
I had thought of exploring a simple rotating queue of users for Beta Cluster a while back to improve isolation, but given the variance of setup/teardown operations across different test suites, it seemed like it still might be difficult to maintain a clean state after a while. Maybe once we have a proper staging cluster and the liberty to wipe the database clean periodically, this type of setup would be worth exploring further.
For now, I think making strides toward more isolated environments and using the API for creation of initial users/pages seems to be the right approach. It's relatively inexpensive—probably the least expensive aspect of browser-driven tests anyway—and can be easily extended in the framework or in test helpers for specific test cases, e.g. creating Flow topics, etc.
Keeping the provisioning of preconditions in code, closer to the test implementation, also makes tests easier to reason about—you known that test case A depends on the precondition that the user that has 3 edits because the test case creates the user and makes 3 edits, instead of having to inspect a preexisting account or rely on comments that may not exist.
We might want something similar for retrieving and locking test articles for editing, but this might not be necessary if our vagrant/virtualization infrastructure is mature enough to ensure tests are isolated.
The environment that's setup for CI browser tests (per patch tests, not the daily runs) is completely isolated for a single project/patchset.[1] I'd like to push this further though, by isolating pages to unique prefixes/namespaces per scenario, so that we might run scenarios in parallel. Another possibility would be to use a multiwiki setup with *n* wikis and simply isolate each subset of scenarios to one of them*.* Of course you'd have to be using our CI infrastructure to benefit from the latter. :P (Couldn't resist ... in all seriousness we could figure out a way to generalize CI-type MW setup as libraries/tools.)
[1] https://www.mediawiki.org/wiki/Continuous_integration/Browser_tests
On Wed, Jul 22, 2015 at 2:51 PM, Dan Duvall dduvall@wikimedia.org wrote:
On Wed, Jul 22, 2015 at 10:43 AM, Brian Gerstle bgerstle@wikimedia.org wrote:
Also, one neat thing I saw done at Spotify was a "Test Data Service." This service's job was to abstract away concrete test data from its qualities. This was mostly used to acquire test users, thus allowing testers to declaratively acquire test data. For example, you could say "I want a free user with X flags" or "I want a premium user who has tracks in their starred playlist." The service's other job was to lock users to prevent parallel tests from corrupting a shared test user's state (also resulting in a false negative).
That's really interesting. Do you have a link to further reading/docs?
Nothing about the the Test Data Service (referred to commonly as TDS) specifically. They're main drum to beat was Model Based Testing http://www.cs.tut.fi/tapahtumat/testaus12/kalvot/Karl_ta.pdf (mostly driven by Kristian Karl). Towards the end of my time there, I was talking to people about creating users on-demand to side-step locking and hand-curating test users, which it seems like you're nudging us towards. I'd be happy to reach out and see if they'd be willing to talk to us more about what they're doing now.
I had thought of exploring a simple rotating queue of users for Beta Cluster a while back to improve isolation, but given the variance of setup/teardown operations across different test suites, it seemed like it still might be difficult to maintain a clean state after a while.
Deadlocks (or missed unlocks) were occasionally a problem, which is why locks were always given a specific life span (e.g. lock for 30 minutes—hopefully your tests finish first!!).
Maybe once we have a proper staging cluster and the liberty to wipe the database clean periodically, this type of setup would be worth exploring further.
For now, I think making strides toward more isolated environments and using the API for creation of initial users/pages seems to be the right approach. It's relatively inexpensive—probably the least expensive aspect of browser-driven tests anyway—and can be easily extended in the framework or in test helpers for specific test cases, e.g. creating Flow topics, etc.
This is where Spotify was going around the time I left. They were very SOA, and trying to push adoption of docker for testing and production. There were some promising results of orchestrating an isolated ecosystem of services on demand, testing them, then tearing it all down. Combine this with pre-baked Docker containers and I think you see where it could lead.
Keeping the provisioning of preconditions in code, closer to the test implementation, also makes tests easier to reason about—you known that test case A depends on the precondition that the user that has 3 edits because the test case creates the user and makes 3 edits, instead of having to inspect a preexisting account or rely on comments that may not exist.
As usual, all about trade-offs. If you want to keep your tests short (ideally you have a lot of them), side-loading or pre-baking service images/containers/etc. with data can be a big time saver.
We might want something similar for retrieving and locking test articles for editing, but this might not be necessary if our vagrant/virtualization infrastructure is mature enough to ensure tests are isolated.
The environment that's setup for CI browser tests (per patch tests, not the daily runs) is completely isolated for a single project/patchset.[1] I'd like to push this further though, by isolating pages to unique prefixes/namespaces per scenario, so that we might run scenarios in parallel. Another possibility would be to use a multiwiki setup with *n* wikis and simply isolate each subset of scenarios to one of them*.* Of course you'd have to be using our CI infrastructure to benefit from the latter. :P (Couldn't resist ... in all seriousness we could figure out a way to generalize CI-type MW setup as libraries/tools.)
[1] https://www.mediawiki.org/wiki/Continuous_integration/Browser_tests
-- Dan Duvall Automation Engineer Wikimedia Foundation http://wikimediafoundation.org
On Wed, Jul 22, 2015 at 9:58 AM, Dan Duvall dduvall@wikimedia.org wrote:
I like this idea for development and manual testing, but I'm not sure it's appropriate for automated testing.
Agreed. I was thinking more for manual testing.
Generally in automated testing, each test case (unit test or integration scenario) should be able to set up or guarantee its preconditions whenever possible—and otherwise communicate its assumptions—which in the context of MediaWiki includes user accounts, user settings, or article content that the test case will manipulate or otherwise depend on. Putting this initial content in MW-Vagrant, which is orthogonal to MW test suites, creates the need for more coupling between test code and test environments and promotes more non-deterministic test behavior, something we've been trying very hard to reduce.[1][2]
Also, for development content, we might want to figure out a better place for it than in the MW-Vagrant repo itself. Article dumps are likely to build up fast, and we risk bloating the project repo itself with big blobs.
[1] https://www.mediawiki.org/wiki/Quality_Assurance/Browser_testing/Environment... [2] https://doc.wikimedia.org/rubygems/mediawiki-selenium/index.html#User_Factor...
On Tue, Jul 21, 2015 at 10:25 AM, Jon Robson jdlrobson@gmail.com wrote:
On 20 Jul 2015 5:56 pm, "Greg Grossmeier" greg@wikimedia.org wrote:
Given the topic, let's keep the QA list in the loop on this so the MW-Vagrant maintainers can participate/see.
Great :)
Also, it looks like the original bug (reported in the MW-Vagrant project) covers this specific request from Reading, no? Essentially,
let's see how far we can get with a general "MW-Vagrant (WMF?) testing data import" instead of a vertical specific "reading-web test data set". If what the Reading team needs is way too much for this then we can break it out, otherwise it seems like a needless distinction.
It does yup. I've already tagged the bug with it. I'm hoping by tackling this we can come up with a common solution. The way I imagine this working in future is we have various vagrant roles for stock data e.g. reading-web-stock-data, editing-web-stock-data, sad-web-stock-data There would also be non team specific stock data that might be a sub role of this, for example, the reading web team commonly has to setup the wikidata role and manually create articles in the wikidata instance and local instance that are tied to each other - this takes a ridiculous amount of time and is one I'm keen to automate, given that we are leaning more heavily on wikidata descriptions and other data in there.
Rob - I've setup https://www.mediawiki.org/wiki/Reading/QA/Sample_articles as a place we can start to collect and think about these pages.
Greg
PS: https://wikitech.wikimedia.org/wiki/Labs_labs_labs
<quote name="Rob Moen" date="2015-07-20" time="17:11:07 -0700"> > Historically developers have had to setup their own content in > mediawiki > and in mediawiki-vagrant. While this can be done with a simple import, > getting everyone on the same page is apparently not as easy. This is > generally problematic as we would like to test code locally and > remotely > with the same content for various reasons. > > Slightly more frustrating, there are pages titled "0.4425590476103759" > on > beta labs. While trying to sign off on a feature, there is usually a > struggle when trying to find an article with suitable content. AFAIK > this > won't change beta labs but would provide a nice standard for our > content on > test wikis. > > We aim to better things by creating a vagrant role for importing a set > of > articles for testing purposes. For more information please see related > phabricator tasks [1] and [2]. > > In hopes of making this a nice collection of articles that multiple > teams > would use, we would like to get input from our designers and devs on > what > types of articles should be in this import. What qualities should > these > articles contain? > > 1: https://phabricator.wikimedia.org/T104561 > 2: https://phabricator.wikimedia.org/T62116 > > > > -- > Rob Moen > Wikimedia Foundation > rmoen@wikimedia.org
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @greg A18D 1138 8E47 FAC8 1C7D |
QA mailing list QA@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/qa
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Dan Duvall Automation Engineer Wikimedia Foundation