Just wanted to send out an update on the progress we made around MW-Vagrant improvements at the Zürich Hackathon. Our primary goal was to make some key production services available in MW-Vagrant in order to make local development/testing easier/more reliable. We made some excellent headway, focussing on a few key services: SSL, Varnish, CentralAuth/Multiwiki.
SSL: I spent a majority of my time focussing on this and received a lot of support/help from Ori. There is now an 'https' role in mw-vagrant which when enabled, will allow you to access your devwiki on port 4430 (forwarded to 443 in Vagrant). There is one outstanding patchset which will make it possible to use $wgSecureLogin in MW-Vagrant: https://gerrit.wikimedia.org/r/#/c/132799/
Varnish: This is proving to be much more difficult than anticipated, however some progress was made and work is ongoing, spearheaded by Andrew Otto. The plan is to set up varnish VCLs for mw-vagrant similar to what is set up for text varnishes in production, with a frontend and backend instance running in vagrant. Andrew is in the midst of refactoring the production varnish module, to make it usable in Vagrant.
CentralAuth/Multiwiki: Bryan Davis, Chris Steipp, and Reedy spent a lot of time hacking on this, and we now have support for multiwiki/CentralAuth in Vagrant! There is still some cleanup work being done for the role to remove kludge/hacks/etc (see https://gerrit.wikimedia.org/r/#/c/132691/).
Also of significant note, Matt Flaschen created a mw-vagrant iso which can be packaged on USB thumb drives, making it possible to set up mw-vagrant without a network connection. There is still some work to be done here to create a one-click installer as well as updating documentation. Matt got this done before the hackathon, and we brought a bunch of USB sticks imaged with the iso, which was instrumental in getting a bunch of folks new to mw-vagrant up and running at the hackathon. This was particularly useful during Bryan Davis's vagrant bootcamp sessions.
I believe Katie Filbert from Wikidata did some mw-vagrant work at the hackathon as well, although I'm not clear on the current status. Katie, can you let us know where things were at with what you were working on?
All in all it felt like a very fruitful hack session, and we're closer than ever to having a ready-to-go developer instance that mimics our production environment. Big thanks to everyone involved in making our work successful.
PS One thing I forgot to mention - mega thanks to WMF's Office IT for helping with ordering and imaging the USB sticks!!!
On Fri, May 16, 2014 at 1:40 PM, Arthur Richards arichards@wikimedia.orgwrote:
Just wanted to send out an update on the progress we made around MW-Vagrant improvements at the Zürich Hackathon. Our primary goal was to make some key production services available in MW-Vagrant in order to make local development/testing easier/more reliable. We made some excellent headway, focussing on a few key services: SSL, Varnish, CentralAuth/Multiwiki.
SSL: I spent a majority of my time focussing on this and received a lot of support/help from Ori. There is now an 'https' role in mw-vagrant which when enabled, will allow you to access your devwiki on port 4430 (forwarded to 443 in Vagrant). There is one outstanding patchset which will make it possible to use $wgSecureLogin in MW-Vagrant: https://gerrit.wikimedia.org/r/#/c/132799/
Varnish: This is proving to be much more difficult than anticipated, however some progress was made and work is ongoing, spearheaded by Andrew Otto. The plan is to set up varnish VCLs for mw-vagrant similar to what is set up for text varnishes in production, with a frontend and backend instance running in vagrant. Andrew is in the midst of refactoring the production varnish module, to make it usable in Vagrant.
CentralAuth/Multiwiki: Bryan Davis, Chris Steipp, and Reedy spent a lot of time hacking on this, and we now have support for multiwiki/CentralAuth in Vagrant! There is still some cleanup work being done for the role to remove kludge/hacks/etc (see https://gerrit.wikimedia.org/r/#/c/132691/).
Also of significant note, Matt Flaschen created a mw-vagrant iso which can be packaged on USB thumb drives, making it possible to set up mw-vagrant without a network connection. There is still some work to be done here to create a one-click installer as well as updating documentation. Matt got this done before the hackathon, and we brought a bunch of USB sticks imaged with the iso, which was instrumental in getting a bunch of folks new to mw-vagrant up and running at the hackathon. This was particularly useful during Bryan Davis's vagrant bootcamp sessions.
I believe Katie Filbert from Wikidata did some mw-vagrant work at the hackathon as well, although I'm not clear on the current status. Katie, can you let us know where things were at with what you were working on?
All in all it felt like a very fruitful hack session, and we're closer than ever to having a ready-to-go developer instance that mimics our production environment. Big thanks to everyone involved in making our work successful.
-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687
On Fri, May 16, 2014 at 3:11 PM, Arthur Richards arichards@wikimedia.org wrote:
PS One thing I forgot to mention - mega thanks to WMF's Office IT for helping with ordering and imaging the USB sticks!!!
On Fri, May 16, 2014 at 1:40 PM, Arthur Richards arichards@wikimedia.orgwrote:
Just wanted to send out an update on the progress we made around MW-Vagrant improvements at the Zürich Hackathon. Our primary goal was to make some key production services available in MW-Vagrant in order to make local development/testing easier/more reliable. We made some excellent headway, focussing on a few key services: SSL, Varnish, CentralAuth/Multiwiki.
SSL: I spent a majority of my time focussing on this and received a lot of support/help from Ori. There is now an 'https' role in mw-vagrant which when enabled, will allow you to access your devwiki on port 4430 (forwarded to 443 in Vagrant). There is one outstanding patchset which will make it possible to use $wgSecureLogin in MW-Vagrant: https://gerrit.wikimedia.org/r/#/c/132799/
Varnish: This is proving to be much more difficult than anticipated, however some progress was made and work is ongoing, spearheaded by Andrew Otto. The plan is to set up varnish VCLs for mw-vagrant similar to what is set up for text varnishes in production, with a frontend and backend instance running in vagrant. Andrew is in the midst of refactoring the production varnish module, to make it usable in Vagrant.
CentralAuth/Multiwiki: Bryan Davis, Chris Steipp, and Reedy spent a lot of time hacking on this, and we now have support for multiwiki/CentralAuth in Vagrant! There is still some cleanup work being done for the role to remove kludge/hacks/etc (see https://gerrit.wikimedia.org/r/#/c/132691/).
Also of significant note, Matt Flaschen created a mw-vagrant iso which can be packaged on USB thumb drives, making it possible to set up mw-vagrant without a network connection. There is still some work to be done here to create a one-click installer as well as updating documentation. Matt got this done before the hackathon, and we brought a bunch of USB sticks imaged with the iso, which was instrumental in getting a bunch of folks new to mw-vagrant up and running at the hackathon. This was particularly useful during Bryan Davis's vagrant bootcamp sessions.
I believe Katie Filbert from Wikidata did some mw-vagrant work at the hackathon as well, although I'm not clear on the current status. Katie, can you let us know where things were at with what you were working on?
All in all it felt like a very fruitful hack session, and we're closer than ever to having a ready-to-go developer instance that mimics our production environment. Big thanks to everyone involved in making our work successful.
I'd like to follow on with a big thanks to Arthur for bringing the topic up and coordinating all of the meetings that led up to the event. Never underestimate the power of having a good organizer on your project!
Bryan
On Fri, May 16, 2014 at 2:18 PM, Bryan Davis bd808@wikimedia.org wrote:
All in all it felt like a very fruitful hack session, and we're closer than ever to having a ready-to-go developer instance that mimics our production environment. Big thanks to everyone involved in making our
work
successful.
I'd like to follow on with a big thanks to Arthur for bringing the topic up and coordinating all of the meetings that led up to the event. Never underestimate the power of having a good organizer on your project!
Yes, definitely agree. Thanks very much, Arthur.
On 05/16/2014 04:40 PM, Arthur Richards wrote:
Just wanted to send out an update on the progress we made around MW-Vagrant improvements at the Zürich Hackathon. Our primary goal was to make some key production services available in MW-Vagrant in order to make local development/testing easier/more reliable. We made some excellent headway, focussing on a few key services: SSL, Varnish, CentralAuth/Multiwiki.
Rock!
Varnish: This is proving to be much more difficult than anticipated, however some progress was made and work is ongoing, spearheaded by Andrew Otto. The plan is to set up varnish VCLs for mw-vagrant similar to what is set up for text varnishes in production, with a frontend and backend instance running in vagrant. Andrew is in the midst of refactoring the production varnish module, to make it usable in Vagrant.
Andrew, I am looking forward to featuring this in https://www.mediawiki.org/wiki/Performance_profiling_for_Wikimedia_code#Star... !
On Fri, May 16, 2014 at 2:40 PM, Arthur Richards arichards@wikimedia.org wrote:
CentralAuth/Multiwiki: Bryan Davis, Chris Steipp, and Reedy spent a lot of time hacking on this, and we now have support for multiwiki/CentralAuth in Vagrant! There is still some cleanup work being done for the role to remove kludge/hacks/etc (see https://gerrit.wikimedia.org/r/#/c/132691/).
The CentralAuth role and the associated puppet config that allows creation of multiple wikis as Apache virtual hosts on a single MediaWiki-Vagrant virtual machine have been merged! Go forth and debug/extend CentralAuth. :)
I'd love to see additional roles created that use the multwiki::wiki Puppet define to add interesting things for testing/debugging like RTL wikis or other complex features such as WikiData that use a collaboration between multiple wikis in the WMF production cluster. If you're interested in working on something like this and get stuck with the Puppet code needed or find shortcomings in the setup that Chris and I developed I'd be glad to try and help work through the issues.
Bryan
Awesome!!!!
On Wed, May 21, 2014 at 9:55 AM, Bryan Davis bd808@wikimedia.org wrote:
On Fri, May 16, 2014 at 2:40 PM, Arthur Richards arichards@wikimedia.org wrote:
CentralAuth/Multiwiki: Bryan Davis, Chris Steipp, and Reedy spent a lot of time hacking on this, and we now have support for multiwiki/CentralAuth in Vagrant! There is still some cleanup work being done for the role to remove
kludge/hacks/etc
The CentralAuth role and the associated puppet config that allows creation of multiple wikis as Apache virtual hosts on a single MediaWiki-Vagrant virtual machine have been merged! Go forth and debug/extend CentralAuth. :)
I'd love to see additional roles created that use the multwiki::wiki Puppet define to add interesting things for testing/debugging like RTL wikis or other complex features such as WikiData that use a collaboration between multiple wikis in the WMF production cluster. If you're interested in working on something like this and get stuck with the Puppet code needed or find shortcomings in the setup that Chris and I developed I'd be glad to try and help work through the issues.
Bryan
Bryan Davis Wikimedia Foundation bd808@wikimedia.org [[m:User:BDavis_(WMF)]] Sr Software Engineer Boise, ID USA irc: bd808 v:415.839.6885 x6855
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On 05/21/2014 12:55 PM, Bryan Davis wrote:
The CentralAuth role and the associated puppet config that allows creation of multiple wikis as Apache virtual hosts on a single MediaWiki-Vagrant virtual machine have been merged! Go forth and debug/extend CentralAuth. :)
This is really great. GettingStarted can now use this, among many other use cases.
Matt
Bryan and Chris, The multiwiki work is fantastic, a big thank you for pursuing this! I tried to use your new module to provide a vagrant development environment for Fundraising's "payments" wiki [1], and I ran up against a large and very solid-looking wall that I think is worth mentioning. We maintain a special release branch of MediaWiki for payments, with a bit of security hardening. We cannot follow trunk development without carefully reading over the new features, and we need to develop against this target so that we catch version incompatibilities before deployment.
I see that multiwiki encapsulates the various wikis by configuration only, and they all share the main codebase. Do you have multiple checkouts of MediaWiki-core on your roadmap, or are we a fringe case? I'd like to help support our development under vagrant, but this issue is a bit of a blocker. Any advice would be appreciated.
Thanks, Adam
[1] https://gerrit.wikimedia.org/r/135326, production is https://payments.wikimedia.org
On Wed, May 21, 2014 at 9:55 AM, Bryan Davis bd808@wikimedia.org wrote:
On Fri, May 16, 2014 at 2:40 PM, Arthur Richards arichards@wikimedia.org wrote:
CentralAuth/Multiwiki: Bryan Davis, Chris Steipp, and Reedy spent a lot of time hacking on this, and we now have support for multiwiki/CentralAuth in Vagrant! There is still some cleanup work being done for the role to remove
kludge/hacks/etc
The CentralAuth role and the associated puppet config that allows creation of multiple wikis as Apache virtual hosts on a single MediaWiki-Vagrant virtual machine have been merged! Go forth and debug/extend CentralAuth. :)
I'd love to see additional roles created that use the multwiki::wiki Puppet define to add interesting things for testing/debugging like RTL wikis or other complex features such as WikiData that use a collaboration between multiple wikis in the WMF production cluster. If you're interested in working on something like this and get stuck with the Puppet code needed or find shortcomings in the setup that Chris and I developed I'd be glad to try and help work through the issues.
Bryan
Bryan Davis Wikimedia Foundation bd808@wikimedia.org [[m:User:BDavis_(WMF)]] Sr Software Engineer Boise, ID USA irc: bd808 v:415.839.6885 x6855
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Thanks Adam!
I'd like to hear more about your exact use case. Getting all of the wikis to run out of a single codebase was a major part of the challenge getting things setup. If you want a repo per wiki, that should be a straightforward case of setting up another apache virtualhost, or using a different subdirectory for each wiki. If you want CentralAuth across them, that's pretty trivial too. Controlling all that with puppet, I defer to bd808 on :)
But it sounds like you may have something else in your use case, let me know.
On Fri, Jun 13, 2014 at 4:09 PM, Adam Wight awight@wikimedia.org wrote:
Bryan and Chris, The multiwiki work is fantastic, a big thank you for pursuing this! I tried to use your new module to provide a vagrant development environment for Fundraising's "payments" wiki [1], and I ran up against a large and very solid-looking wall that I think is worth mentioning. We maintain a special release branch of MediaWiki for payments, with a bit of security hardening. We cannot follow trunk development without carefully reading over the new features, and we need to develop against this target so that we catch version incompatibilities before deployment.
I see that multiwiki encapsulates the various wikis by configuration only, and they all share the main codebase. Do you have multiple checkouts of MediaWiki-core on your roadmap, or are we a fringe case? I'd like to help support our development under vagrant, but this issue is a bit of a blocker. Any advice would be appreciated.
Thanks, Adam
[1] https://gerrit.wikimedia.org/r/135326, production is https://payments.wikimedia.org
On Wed, May 21, 2014 at 9:55 AM, Bryan Davis bd808@wikimedia.org wrote:
On Fri, May 16, 2014 at 2:40 PM, Arthur Richards arichards@wikimedia.org wrote:
CentralAuth/Multiwiki: Bryan Davis, Chris Steipp, and Reedy spent a lot of time hacking on this, and we now have support for multiwiki/CentralAuth in Vagrant! There is still some cleanup work being done for the role to remove
kludge/hacks/etc
The CentralAuth role and the associated puppet config that allows creation of multiple wikis as Apache virtual hosts on a single MediaWiki-Vagrant virtual machine have been merged! Go forth and debug/extend CentralAuth. :)
I'd love to see additional roles created that use the multwiki::wiki Puppet define to add interesting things for testing/debugging like RTL wikis or other complex features such as WikiData that use a collaboration between multiple wikis in the WMF production cluster. If you're interested in working on something like this and get stuck with the Puppet code needed or find shortcomings in the setup that Chris and I developed I'd be glad to try and help work through the issues.
Bryan
Bryan Davis Wikimedia Foundation bd808@wikimedia.org [[m:User:BDavis_(WMF)]] Sr Software Engineer Boise, ID USA irc: bd808 v:415.839.6885 x6855
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Adam,
I wanted to avoid the complexity of a full multi version setup if we could, but if there are more than a couple of roles that would benefit from such features it would be possible. The easiest thing for your payment wiki may be to follow the pattern of the wikimania_scholarships role and add your own apache vhost and git checkout management. I'd be glad to help you work on this if you'd like some assistance. We might be able to add a bit more configurability to some of the existing puppet classes and defines to make working with multiple checkouts of mw-core easier.
Bryan
On Friday, June 13, 2014, Adam Wight awight@wikimedia.org wrote:
Bryan and Chris, The multiwiki work is fantastic, a big thank you for pursuing this! I tried to use your new module to provide a vagrant development environment for Fundraising's "payments" wiki [1], and I ran up against a large and very solid-looking wall that I think is worth mentioning. We maintain a special release branch of MediaWiki for payments, with a bit of security hardening. We cannot follow trunk development without carefully reading over the new features, and we need to develop against this target so that we catch version incompatibilities before deployment.
I see that multiwiki encapsulates the various wikis by configuration only, and they all share the main codebase. Do you have multiple checkouts of MediaWiki-core on your roadmap, or are we a fringe case? I'd like to help support our development under vagrant, but this issue is a bit of a blocker. Any advice would be appreciated.
Thanks, Adam
[1] https://gerrit.wikimedia.org/r/135326, production is https://payments.wikimedia.org
On Wed, May 21, 2014 at 9:55 AM, Bryan Davis <bd808@wikimedia.org javascript:;> wrote:
On Fri, May 16, 2014 at 2:40 PM, Arthur Richards <arichards@wikimedia.org javascript:;> wrote:
CentralAuth/Multiwiki: Bryan Davis, Chris Steipp, and Reedy spent a lot of time hacking on
this,
and we now have support for multiwiki/CentralAuth in Vagrant! There is still some cleanup work being done for the role to remove
kludge/hacks/etc
The CentralAuth role and the associated puppet config that allows creation of multiple wikis as Apache virtual hosts on a single MediaWiki-Vagrant virtual machine have been merged! Go forth and debug/extend CentralAuth. :)
I'd love to see additional roles created that use the multwiki::wiki Puppet define to add interesting things for testing/debugging like RTL wikis or other complex features such as WikiData that use a collaboration between multiple wikis in the WMF production cluster. If you're interested in working on something like this and get stuck with the Puppet code needed or find shortcomings in the setup that Chris and I developed I'd be glad to try and help work through the issues.
Bryan
Bryan Davis Wikimedia Foundation <bd808@wikimedia.org
[[m:User:BDavis_(WMF)]] Sr Software Engineer Boise, ID USA irc: bd808 v:415.839.6885 x6855
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org javascript:; https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org javascript:; https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Bryan,
I think I need to take you up on the offer to help. I can do the coding, but I need some borrowed insight to get started. I don't think the wikimania_scholarships model is a good one to follow, I'd much rather add configurability to the mediawiki::wiki or multiwiki::wiki classes. Unfortunately, I see a lot of cascading changes being necessary, which makes me think I'm on the wrong track.
-Adam
On Fri, Jun 13, 2014 at 5:29 PM, Bryan Davis bd808@wikimedia.org wrote:
Adam,
I wanted to avoid the complexity of a full multi version setup if we could, but if there are more than a couple of roles that would benefit from such features it would be possible. The easiest thing for your payment wiki may be to follow the pattern of the wikimania_scholarships role and add your own apache vhost and git checkout management. I'd be glad to help you work on this if you'd like some assistance. We might be able to add a bit more configurability to some of the existing puppet classes and defines to make working with multiple checkouts of mw-core easier.
Bryan
On Friday, June 13, 2014, Adam Wight awight@wikimedia.org wrote:
Bryan and Chris, The multiwiki work is fantastic, a big thank you for pursuing this! I tried to use your new module to provide a vagrant development environment for Fundraising's "payments" wiki [1], and I ran up against a large and very solid-looking wall that I think is worth mentioning. We maintain a special release branch of MediaWiki for payments, with a bit of security hardening. We cannot follow trunk development without carefully reading over the new features, and we need to develop against this target so that we catch version incompatibilities before deployment.
I see that multiwiki encapsulates the various wikis by configuration only, and they all share the main codebase. Do you have multiple checkouts of MediaWiki-core on your roadmap, or are we a fringe case? I'd like to help support our development under vagrant, but this issue is a bit of a blocker. Any advice would be appreciated.
Thanks, Adam
[1] https://gerrit.wikimedia.org/r/135326, production is https://payments.wikimedia.org
On Wed, May 21, 2014 at 9:55 AM, Bryan Davis bd808@wikimedia.org wrote:
On Fri, May 16, 2014 at 2:40 PM, Arthur Richards arichards@wikimedia.org wrote:
CentralAuth/Multiwiki: Bryan Davis, Chris Steipp, and Reedy spent a lot of time hacking on
this,
and we now have support for multiwiki/CentralAuth in Vagrant! There is still some cleanup work being done for the role to remove
kludge/hacks/etc
The CentralAuth role and the associated puppet config that allows creation of multiple wikis as Apache virtual hosts on a single MediaWiki-Vagrant virtual machine have been merged! Go forth and debug/extend CentralAuth. :)
I'd love to see additional roles created that use the multwiki::wiki Puppet define to add interesting things for testing/debugging like RTL wikis or other complex features such as WikiData that use a collaboration between multiple wikis in the WMF production cluster. If you're interested in working on something like this and get stuck with the Puppet code needed or find shortcomings in the setup that Chris and I developed I'd be glad to try and help work through the issues.
Bryan
Bryan Davis Wikimedia Foundation bd808@wikimedia.org [[m:User:BDavis_(WMF)]] Sr Software Engineer Boise, ID USA irc: bd808 v:415.839.6885 x6855
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
-- Bryan Davis Wikimedia Foundation bd808@wikimedia.org [[m:User:BDavis_(WMF)]] Sr Software Engineer Boise, ID USA irc: bd808 v:415.839.6885 x6855
On Mon, Jul 7, 2014 at 4:38 PM, Adam Wight awight@wikimedia.org wrote:
Bryan,
I think I need to take you up on the offer to help. I can do the coding, but I need some borrowed insight to get started. I don't think the wikimania_scholarships model is a good one to follow, I'd much rather add configurability to the mediawiki::wiki or multiwiki::wiki classes. Unfortunately, I see a lot of cascading changes being necessary, which makes me think I'm on the wrong track.
I'd love to help. Before we go too far down the multiple MW checkouts path, would it be possible for your testing to have all wikis on the Vagrant guest machine using the special branch? If that's not possible I have a few other partially formed ideas of how we might be able to make this work. I'm not sure that I'll have a lot of time to work on it in next few weeks, but I could work on it during the Wikimania Hackathon as it would be in general an interesting feature to support for mw-vagrant.
Bryan
wikitech-l@lists.wikimedia.org