I am writing to announce mediawiki-containers [1], a simple installer for MediaWiki with VisualEditor, Parsoid, RESTBase and other services, using Linux containers.
The main goal of this project is to make it really easy to set up and maintain a fully featured MediaWiki system on a wide range of platforms. The project is in an early stage, but already supports full installation on Ubuntu, Debian and other systemd-based distributions, as well as starting containers on OS X via the Docker toolbox [6].
These are the basic steps involved in setting up your own MediaWiki instance with VisualEditor:
1) Get a Linux VM in labs or from a hosting provider, and select Debian (Jessie or newer) or Ubuntu 15.04+ as the distribution. Commercial VMs with reasonable specifications cost about $5 per month [2]. 2) Log into your VM, and run this command [7]: curl https://raw.githubusercontent.com/wikimedia/mediawiki-containers/master/medi... | sudo bash 3) Answer the questions in the installer.
Here is a screencast of an installer run, illustrating steps 2) and 3): https://people.wikimedia.org/~gwicke/mediawiki-containers-install.ogv
Under the hood, mediawiki-containers uses several Docker containers: - wikimedia/mediawiki [3] with MediaWiki 1.27-wmf9 and VisualEditor. - wikimedia/mediawiki-node-services [4] with Parsoid and RESTBase running in a single process to minimize memory use. - MariaDB as the database backend [5].
Data and configurations are stored on the host system in /srv/mediawiki-containers/data, which means that upgrading is as simple as fetching the latest container images by re-running the installer. Optionally, the installer can set up automated nightly updates, which helps to keep your wiki installation up to date.
The project is brand new, so there is a fair chance that you will encounter bugs. Please report issues at https://phabricator.wikimedia.org/maniphest/task/create/?projects=mediawiki-... .
Here are some ideas we have for the next steps:
- Forward `/api/rest_v1/` to RESTBase & configure RESTBase updates. Enable Wikitext / HTML switching in VE. - Improve security: - Run each container under a different, unprivileged user. - Secure the install / update process with signatures. - Add popular extensions, and streamline the support for custom extensions. - Add services like mathoid, graphoid. - Use the HHVM PHP runtime instead of Zend, possibly using ideas from https://github.com/kasperisager/php-dockerized. - Support developer use cases: - Optionally mount code volumes from the host system. - Improve configuration customization support. - Support for more distributions.
Let us know what you think & what you would like to see next at https://phabricator.wikimedia.org/T92826.
Happy holidays,
Gabriel Wicke and the Services team
[1]: https://github.com/wikimedia/mediawiki-containers [2]: http://serverbear.com/compare?Sort=BearScore&Order=desc&Server+Type=... [3]: https://github.com/wikimedia/mediawiki-docker and https://hub.docker.com/r/wikimedia/mediawiki [4]: https://github.com/wikimedia/mediawiki-node-services and https://hub.docker.com/r/wikimedia/mediawiki-node-services/ [5]: https://hub.docker.com/_/mariadb/ [6]: https://docs.docker.com/mac/step_one/ [7]: We agree that `curl | bash` has its risks, but it is hard to beat for simplicity. The Chef project has a good discussion of pros, cons & alternatives at https://www.chef.io/blog/2015/07/16/5-ways-to-deal-with-the-install-sh-curl-... .
On Thursday, December 24, 2015 03:57:50 PM Gabriel Wicke wrote:
I am writing to announce mediawiki-containers [1], a simple installer for MediaWiki with VisualEditor, Parsoid, RESTBase and other services, using Linux containers.
Hi Gabriel,
we at WikiToLearn have automated all of our infrastructure set-up with docker containers. Our scripts and Dockerfiles are public and on github (waiting to be moved on git.kde.org...), maybe we can do something together? :-)
We have a full Mediawiki stack, including OCG, memcached and Parsoid, and it has been working like a breeze on many platforms. How can we cooperate?
Bye, -Riccardo
Hi Riccardo,
On Thu, Dec 24, 2015 at 4:02 PM, Riccardo Iaconelli riccardo@kde.org wrote:
We have a full Mediawiki stack, including OCG, memcached and Parsoid, and it has been working like a breeze on many platforms.
I'm glad to hear that Docker has worked well for you. I wasn't aware of your images, so thanks for the pointers!
How can we cooperate?
The focus of mediawiki-containers is currently primarily on small installs with limited hardware resources, but I think that we can make the core flexible enough to support an ecosystem of optional features and services like OCG.
Your input on what you need out of the base mediawiki image would be much appreciated. I think mediawiki-docker [1] already covers much of what you are doing in https://github.com/WikiToLearn/WebSrv, but I also see some bits like X-Forwarded-For handling that are still missing. What is your view on switching to HHVM?
Another obvious opportunity is to integrate your work on OCG as an optional feature in the installer. The multi-image setup in mediawiki-containers should make this relatively straightforward.
Gabriel
On Thu, Dec 24, 2015 at 3:57 PM, Gabriel Wicke gwicke@wikimedia.org wrote:
I am writing to announce mediawiki-containers [1], a simple installer for MediaWiki with VisualEditor, Parsoid, RESTBase and other services, using Linux containers.
This is very nice work -- kudos. Is it too soon to envision running this (or rather, some future iteration) in production at Wikimedia? What would need to happen?
On 27 December 2015 at 11:52, Ori Livneh ori@wikimedia.org wrote:
On Thu, Dec 24, 2015 at 3:57 PM, Gabriel Wicke gwicke@wikimedia.org wrote:
I am writing to announce mediawiki-containers [1], a simple installer for MediaWiki with VisualEditor, Parsoid, RESTBase and other services, using Linux containers.
This is very nice work -- kudos. Is it too soon to envision running this (or rather, some future iteration) in production at Wikimedia? What would need to happen?
Ping on this. I for one would be interested too. :-)
J.
I see containers as one ingredient in a more automated and tested pipeline from development through CI to production. mediawiki-containers could expand to cover the development use case, but I think we can and should move from there into CI, and finally production.
Right now, Yuvi is evaluating the Kubernetes cluster manager in labs. Its features include scheduling of "pods" (groups of containers) to hardware nodes, networking, rolling deploys and more. While all these features provide a very high degree of automation, they also mean that failures in Kubernetes can have grave consequences. I think operations are wise to wait for Kubernetes to mature a bit further before considering it for critical production use cases.
Rather than waiting until one-stop cluster managers are mature, we could also start with a more traditional config / deploy system. I have played with this approach using Ansible [1] a while ago, and the ergonomics are pretty much the same as git-based deploys. There is also some support to run docker images in systemd, which could be an alternative if we want to avoid the dependency on the docker runtime in production. This older task lists some options: https://phabricator.wikimedia.org/T93439
Lets get together and figure out a plan.
Gabriel
[1]: http://docs.ansible.com/ansible/docker_module.html
On Fri, Jan 29, 2016 at 8:23 PM, James Forrester jforrester@wikimedia.org wrote:
On 27 December 2015 at 11:52, Ori Livneh ori@wikimedia.org wrote:
On Thu, Dec 24, 2015 at 3:57 PM, Gabriel Wicke gwicke@wikimedia.org wrote:
I am writing to announce mediawiki-containers [1], a simple installer for MediaWiki with VisualEditor, Parsoid, RESTBase and other services, using Linux containers.
This is very nice work -- kudos. Is it too soon to envision running this (or rather, some future iteration) in production at Wikimedia? What would need to happen?
Ping on this. I for one would be interested too. :-)
J.
James D. Forrester Lead Product Manager, Editing Wikimedia Foundation, Inc.
jforrester@wikimedia.org | @jdforrester
On Sat, Jan 30, 2016 at 9:59 AM, Gabriel Wicke gwicke@wikimedia.org wrote:
Right now, Yuvi is evaluating the Kubernetes cluster manager in labs.
Just a clarification: Yuvi has already evaluated kubernetes and it's being actively used to build an awesome replacement for at least part of what toollabs does right now. A handful of tools are already running, with success, on it for quite a long time.
Its features include scheduling of "pods" (groups of containers) to hardware nodes, networking, rolling deploys and more. While all these features provide a very high degree of automation, they also mean that failures in Kubernetes can have grave consequences. I think operations are wise to wait for Kubernetes to mature a bit further before considering it for critical production use cases.
Failures in any complex system are surely scary, but kubernetes seems stable enough to be evaluated for production use. We also had an unconference session at the WMDS about this - or better what we want to achieve by using kubernetes as a tool.
I will also stress that there are more "mature" cluster/container framework like Apache Mesos/Aurora/Marathon, but after taking a hard look at them me and Yuvi evaluated that kubernetes is way more promising for any of our use cases.
This is still a bit further away in the future, anyways. There is already a phabricator task for this, which is anyways sitting idle at the moment as it's not in our immediate roadmap. The task is by the way trying to be independent of the specific technology in describing what we actually want to achieve. Kubernetes, as any other product we might use, is just a mean to an end, and we should never be in love with any specific technology.
https://phabricator.wikimedia.org/T122822
There is also some support to run docker images in systemd, which could be an alternative if we want to avoid the dependency on the docker runtime in production.
I guess you mean containers can run within systemd, but I don't think just running containers instead of firejail would give us any practical advantage at the moment from any operational prespective, but I might miss the point.
Lets get together and figure out a plan.
Let's do it! maybe next quarter when ops are not mostly focused on the datacenter switch it will be easier, I guess :)
Cheers,
Giuseppe
wikitech-l@lists.wikimedia.org