When I was hired as QA Lead almost seven months ago, WMF lacked a test environment where
* code was routinely deployed ahead of production * the test environment emulated the production environment closely * aspects of the test environment (config, permissions, etc.) could be easily and reliably manipulated for testing purposes
Today I am happy to announce that beta labs fulfills those needs.
Beta labs is intended to host the upcoming release of Mediawiki, plus those extensions scheduled for deployment to production, for the purpose of testing and investigation.
As of a little while ago, Mediawiki, AFTv5, New Pages Feed/Page Curation, and UploadWizard are being deployed to beta labs from git automatically and reliably. The configurations for those extensions are also being managed in git. The environment itself is managed via puppet, and emulates production to the greatest extent possible. Many many thanks to Antoine Musso for making this possible.
As of this week, all these extensions are up, running, and configured to be useful. Note that they are not perfect, just useful. For example, right now on beta enwiki both AFTv4 and AFTv5 input forms appear on the same page in many cases, because I was experimenting with what happens when these extensions are not configured correctly. Some actions from the Page Curation toolbar never complete. As these glitches become important to testing, we will get them working correctly, and likely will find out some interesting things about the software along the way.
The timing for this announcement is excellent, because new QA Engineers will be joining WMF soon (more on that next week), and beta labs will be a prime target for the browser-level end-to-end automated tests we will shortly be creating. Also, we have been wanting to retire the 'prototype' host for some time, and having AFTv5 etc. on beta labs should make that possible.
In summary, beta labs is up and running with current code for Mediawiki and critical extensions, and at this point the best way to improve beta labs is to use it.
http://en.wikipedia.beta.wmflabs.org/wiki/Special:ArticleFeedbackv5 http://en.wikipedia.beta.wmflabs.org/wiki/Special:NewPagesFeed http://commons.wikimedia.org/wiki/Special:UploadWizard
http://commons.wikimedia.beta.wmflabs.org/wiki/Special:UploadWizard
I am a bad editor. -Chris
On Fri, Sep 28, 2012 at 9:48 AM, Chris McMahon cmcmahon@wikimedia.orgwrote:
When I was hired as QA Lead almost seven months ago, WMF lacked a test environment where
- code was routinely deployed ahead of production
- the test environment emulated the production environment closely
- aspects of the test environment (config, permissions, etc.) could be
easily and reliably manipulated for testing purposes
Today I am happy to announce that beta labs fulfills those needs.
Beta labs is intended to host the upcoming release of Mediawiki, plus those extensions scheduled for deployment to production, for the purpose of testing and investigation.
As of a little while ago, Mediawiki, AFTv5, New Pages Feed/Page Curation, and UploadWizard are being deployed to beta labs from git automatically and reliably. The configurations for those extensions are also being managed in git. The environment itself is managed via puppet, and emulates production to the greatest extent possible. Many many thanks to Antoine Musso for making this possible.
As of this week, all these extensions are up, running, and configured to be useful. Note that they are not perfect, just useful. For example, right now on beta enwiki both AFTv4 and AFTv5 input forms appear on the same page in many cases, because I was experimenting with what happens when these extensions are not configured correctly. Some actions from the Page Curation toolbar never complete. As these glitches become important to testing, we will get them working correctly, and likely will find out some interesting things about the software along the way.
The timing for this announcement is excellent, because new QA Engineers will be joining WMF soon (more on that next week), and beta labs will be a prime target for the browser-level end-to-end automated tests we will shortly be creating. Also, we have been wanting to retire the 'prototype' host for some time, and having AFTv5 etc. on beta labs should make that possible.
In summary, beta labs is up and running with current code for Mediawiki and critical extensions, and at this point the best way to improve beta labs is to use it.
http://en.wikipedia.beta.wmflabs.org/wiki/Special:ArticleFeedbackv5 http://en.wikipedia.beta.wmflabs.org/wiki/Special:NewPagesFeed http://commons.wikimedia.org/wiki/Special:UploadWizard
Chris, this sounds really cool. Can you point us to some specs about how the test environment is set up (what is the architecture like, what services are running, etc)? How closely does it emulate the production environment? Does the beta labs environment provide load balanced squid/varnish caching layers, configured similarly to the produciton cluster? If not, is that something we can hope to see? Is the setup something that we can package up and easily deploy to new instances in labs?
One of the biggest challenges I've had testing code is not being able to easily test what will happen when requests are handled by squid or varnish (particularly for mobile where request routing is more complicated) - to the point where there are something we kinda just have to test in production. I cannot wait until this is no longer the case :)
Also, how can other projects/extensions start getting automatically pushed to the beta labs setup?
Thanks! Arthur
On Fri, Sep 28, 2012 at 8:48 AM, Chris McMahon cmcmahon@wikimedia.orgwrote:
When I was hired as QA Lead almost seven months ago, WMF lacked a test environment where
- code was routinely deployed ahead of production
- the test environment emulated the production environment closely
- aspects of the test environment (config, permissions, etc.) could be
easily and reliably manipulated for testing purposes
Today I am happy to announce that beta labs fulfills those needs.
Beta labs is intended to host the upcoming release of Mediawiki, plus those extensions scheduled for deployment to production, for the purpose of testing and investigation.
As of a little while ago, Mediawiki, AFTv5, New Pages Feed/Page Curation, and UploadWizard are being deployed to beta labs from git automatically and reliably. The configurations for those extensions are also being managed in git. The environment itself is managed via puppet, and emulates production to the greatest extent possible. Many many thanks to Antoine Musso for making this possible.
As of this week, all these extensions are up, running, and configured to be useful. Note that they are not perfect, just useful. For example, right now on beta enwiki both AFTv4 and AFTv5 input forms appear on the same page in many cases, because I was experimenting with what happens when these extensions are not configured correctly. Some actions from the Page Curation toolbar never complete. As these glitches become important to testing, we will get them working correctly, and likely will find out some interesting things about the software along the way.
The timing for this announcement is excellent, because new QA Engineers will be joining WMF soon (more on that next week), and beta labs will be a prime target for the browser-level end-to-end automated tests we will shortly be creating. Also, we have been wanting to retire the 'prototype' host for some time, and having AFTv5 etc. on beta labs should make that possible.
In summary, beta labs is up and running with current code for Mediawiki and critical extensions, and at this point the best way to improve beta labs is to use it.
http://en.wikipedia.beta.wmflabs.org/wiki/Special:ArticleFeedbackv5 http://en.wikipedia.beta.wmflabs.org/wiki/Special:NewPagesFeed http://commons.wikimedia.org/wiki/Special:UploadWizard _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Fri, Sep 28, 2012 at 11:10 AM, Arthur Richards arichards@wikimedia.orgwrote:
Chris, this sounds really cool. Can you point us to some specs about how the test environment is set up (what is the architecture like, what services are running, etc)? How closely does it emulate the production environment? Does the beta labs environment provide load balanced squid/varnish caching layers, configured similarly to the produciton cluster? If not, is that something we can hope to see? Is the setup something that we can package up and easily deploy to new instances in labs?
This is somewhat out of date and in the queue to be updated, so take it with a grain of salt, but most of that is documented here: https://labsconsole.wikimedia.org/wiki/Deployment/Overview
I have to admit I am still learning my way around, I'll defer to Antoine for any detailed answers on the configuration.
Also, how can other projects/extensions start getting automatically pushed to the beta labs setup?
The rule of thumb so far is that any code with a scheduled deployment date may (and probably should) be deployed to beta labs beforehand. In practice this has so far meant AFTv5, NewPagesFeed + Curation Toolbar, and to some extent TimedMediaHandler (testing TMH was the original motivation to get this environment in place). For example, AFTv5 is about to undergo some back end changes, I want to have an automated end-to-end test in place for it to be sure the front end is not changed by accident.
This is still early days though, if you have a project that could benefit from the beta labs test env, I'm open to discussing anything along those lines.
This is super awesome.
This is something that could be useful for MobileFrontend, although I suspect there will need to be some additional configuration work to mimic how mobile requests get handled on the cluster. I'll ping Antoine :)
On Fri, Sep 28, 2012 at 10:30 AM, Chris McMahon cmcmahon@wikimedia.orgwrote:
On Fri, Sep 28, 2012 at 11:10 AM, Arthur Richards arichards@wikimedia.orgwrote:
Chris, this sounds really cool. Can you point us to some specs about how the test environment is set up (what is the architecture like, what services are running, etc)? How closely does it emulate the production environment? Does the beta labs environment provide load balanced squid/varnish caching layers, configured similarly to the produciton cluster? If not, is that something we can hope to see? Is the setup something that we can package up and easily deploy to new instances in labs?
This is somewhat out of date and in the queue to be updated, so take it with a grain of salt, but most of that is documented here: https://labsconsole.wikimedia.org/wiki/Deployment/Overview
I have to admit I am still learning my way around, I'll defer to Antoine for any detailed answers on the configuration.
Also, how can other projects/extensions start getting automatically
pushed
to the beta labs setup?
The rule of thumb so far is that any code with a scheduled deployment date may (and probably should) be deployed to beta labs beforehand. In practice this has so far meant AFTv5, NewPagesFeed + Curation Toolbar, and to some extent TimedMediaHandler (testing TMH was the original motivation to get this environment in place). For example, AFTv5 is about to undergo some back end changes, I want to have an automated end-to-end test in place for it to be sure the front end is not changed by accident.
This is still early days though, if you have a project that could benefit from the beta labs test env, I'm open to discussing anything along those lines. _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Fri, Sep 28, 2012 at 11:37 AM, Arthur Richards arichards@wikimedia.orgwrote:
This is super awesome.
This is something that could be useful for MobileFrontend, although I suspect there will need to be some additional configuration work to mimic how mobile requests get handled on the cluster. I'll ping Antoine :)
Also, we'll have a Mobile QA person hired Real Soon Now, so more motivation to get things like this in place.
On Fri, Sep 28, 2012 at 10:43 AM, Chris McMahon cmcmahon@wikimedia.org wrote:
On Fri, Sep 28, 2012 at 11:37 AM, Arthur Richards arichards@wikimedia.orgwrote:
This is super awesome.
This is something that could be useful for MobileFrontend, although I suspect there will need to be some additional configuration work to mimic how mobile requests get handled on the cluster. I'll ping Antoine :)
Also, we'll have a Mobile QA person hired Real Soon Now, so more motivation to get things like this in place.
A few questions...
I love that we now finally have a way to test "production changes on a non-production environment".
1. How can I get access to this environment, so I can fiddle with it, too (or is this done through "test puppet changes" or something? 2. Is it possible to simply clone another one of these environments, and if so, how is that done?
Cheers!
On Fri, Sep 28, 2012 at 11:49 AM, Siebrand Mazeland (WMF) < smazeland@wikimedia.org> wrote:
- How can I get access to this environment, so I can fiddle with it,
too (or is this done through "test puppet changes" or something?
Depends on what you mean by 'access' and 'fiddle'.
Beta labs is configured from the same git project as production, except that there is a CommonSettings-wmflabs.php file, db-wmflabs.php file etc. that override the production settings where desired. For example, you can see my recent commit to CommonSettings-wmflabs.php from two days ago that enabled AFTv5 for 100% of beta enwiki in the queue here: https://gerrit.wikimedia.org/r/gitweb?p=operations%2Fmediawiki-config.git;a=...
- Is it possible to simply clone another one of these environments,
and if so, how is that done?
My understanding is that there are still a few live hacks in place yet to be managed, but since the vast majority of the environment is managed from puppet and the config is in git, that should be possible. I'm not sure of the mechanism by which to accomplish it, though. Cloning beta labs is not a topic we've considered so far.
On Fri, Sep 28, 2012 at 10:49 AM, Siebrand Mazeland (WMF) < smazeland@wikimedia.org> wrote:
- How can I get access to this environment, so I can fiddle with it,
too (or is this done through "test puppet changes" or something? 2. Is it possible to simply clone another one of these environments, and if so, how is that done?
beta.wmflabs is a complex multi-machine setup. Read https://labsconsole.wikimedia.org/wiki/Deployment/Overview (poorly named, it's just for the beta).
However, it would be extremely useful to have a way to make a developer's local MediaWiki or small instance "much like" production. In my case I keep adding more extensions and settings to my local install every time I find something that affects behavior, and I'm still bitten by changes like turning on $wgHtml5. I couldn't figure out how to "import" the production InitialiseSettings.php and CommonSettings.php to a simple single instance.
People seem interested in this, meanwhile it seems everyone's figuring it out for themselves. As a start, I added more to https://labsconsole.wikimedia.org/wiki/Help:InstanceConfigMediawiki and on its talk page mentioned how E3 sets up its MW instances. Please share what you're doing there.
In the promised land, developers use vagrant to run local VM instances on laptops that puppet configures to run a production-ish MediaWiki. At Etsy and Facebook, the day a developer walks in she can make changes in her personal VM and push them to production (or so they claim in blog posts and meetups ;-) ).
On Fri, Sep 28, 2012 at 8:48 AM, Chris McMahon cmcmahon@wikimedia.org wrote:
When I was hired as QA Lead almost seven months ago, WMF lacked a test environment where
- code was routinely deployed ahead of production
- the test environment emulated the production environment closely
- aspects of the test environment (config, permissions, etc.) could be
easily and reliably manipulated for testing purposes
Today I am happy to announce that beta labs fulfills those needs.
This is excellent news. Congrats to all involved!
- Ryan
wikitech-l@lists.wikimedia.org