Hi all,
Currently I have five wikis with the largest one being about 35k articles (109k pages) and pretty heavily trafficked. My basic server architecture is four web servers behind a load balancer and with a single NFS server that shares out a directory that contains the upload directory content for each of the five wikis, e.g. /wiki/wiki1, /wiki/wiki2, etc. (There are also MySQL and Memcached servers but they are not relevant to this discussion.) Each web server mounts /wiki in one location, say /var/www/images and each of the five MediaWiki instances on the server has its images subdirectory as a symlink to its corresponding subdirectory under the mount, e.g. /var/www/images/wiki2.
Obviously the NFS server is a single point of failure but I've yet to come up with a good alternative shared-filesystem architecture that doesn't require an expensive license like SNFS.
Finally, I'm considering moving the whole shebang to AWS but using S3 directly on the web servers doesn't seem viable in this architecture.
So I'm wondering how others are approaching the design of load balancing (multiple instances of) MediaWiki across multiple web servers while maintaining a single source for each wikis upload directory content. I'm willing to COMPLETELY reevaluate my wiki server architecture as long as it's fast and highly available, so all suggestions are welcome!
Justin
You are mentioning NFS why not use rsync to replicate to a 2ndary nfs server and set it to run lets say every 5 to 10 min or how ever often you want to keep the 2ndary server updated.
On Tue, Oct 28, 2014 at 1:13 AM, Justin Lloyd jclbugz@gmail.com wrote:
Hi all,
Currently I have five wikis with the largest one being about 35k articles (109k pages) and pretty heavily trafficked. My basic server architecture is four web servers behind a load balancer and with a single NFS server that shares out a directory that contains the upload directory content for each of the five wikis, e.g. /wiki/wiki1, /wiki/wiki2, etc. (There are also MySQL and Memcached servers but they are not relevant to this discussion.) Each web server mounts /wiki in one location, say /var/www/images and each of the five MediaWiki instances on the server has its images subdirectory as a symlink to its corresponding subdirectory under the mount, e.g. /var/www/images/wiki2.
Obviously the NFS server is a single point of failure but I've yet to come up with a good alternative shared-filesystem architecture that doesn't require an expensive license like SNFS.
Finally, I'm considering moving the whole shebang to AWS but using S3 directly on the web servers doesn't seem viable in this architecture.
So I'm wondering how others are approaching the design of load balancing (multiple instances of) MediaWiki across multiple web servers while maintaining a single source for each wikis upload directory content. I'm willing to COMPLETELY reevaluate my wiki server architecture as long as it's fast and highly available, so all suggestions are welcome!
Justin _______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
I think my explanation was not the clearest it could have been. Let's say for the moment that I have one wiki. That wiki is served by a load balancer in front of a server farm consisting of four Apache vhosts, one per physical server, each with its own copy of MediaWiki, LocalSettings.php, etc. Thus, a request for say http://wiki.domain.com/wiki/Main_Page (and thus all of its included images, css, js, etc.) is actually distributed by the load balancer across the four vhosts. Each of the four physical hosts NFS mounts the same shared directory from the single NFS server so that all four Apache vhosts have simultaneous read-write access to the same uploaded multimedia content.
In case that description is missing your point, I'll add that I do indeed rsync the NFS server's shared directory to another server nightly (I could easily shorten that interval), which in turn gets rsynced to an offsite server. So the NFS server is a single point of failure, but I do have both local and remote copies of the uploaded content. My desire is for increased reliability of that backend fileserver.
Does that answer your question or am I still missing your point? :)
Justin
On Mon, Oct 27, 2014 at 10:41 PM, Jonathan Aquilina eagles051387@gmail.com wrote:
You are mentioning NFS why not use rsync to replicate to a 2ndary nfs server and set it to run lets say every 5 to 10 min or how ever often you want to keep the 2ndary server updated.
On Tue, Oct 28, 2014 at 1:13 AM, Justin Lloyd jclbugz@gmail.com wrote:
Hi all,
Currently I have five wikis with the largest one being about 35k articles (109k pages) and pretty heavily trafficked. My basic server architecture
is
four web servers behind a load balancer and with a single NFS server that shares out a directory that contains the upload directory content for
each
of the five wikis, e.g. /wiki/wiki1, /wiki/wiki2, etc. (There are also MySQL and Memcached servers but they are not relevant to this
discussion.)
Each web server mounts /wiki in one location, say /var/www/images and
each
of the five MediaWiki instances on the server has its images subdirectory as a symlink to its corresponding subdirectory under the mount, e.g. /var/www/images/wiki2.
Obviously the NFS server is a single point of failure but I've yet to
come
up with a good alternative shared-filesystem architecture that doesn't require an expensive license like SNFS.
Finally, I'm considering moving the whole shebang to AWS but using S3 directly on the web servers doesn't seem viable in this architecture.
So I'm wondering how others are approaching the design of load balancing (multiple instances of) MediaWiki across multiple web servers while maintaining a single source for each wikis upload directory content. I'm willing to COMPLETELY reevaluate my wiki server architecture as long as it's fast and highly available, so all suggestions are welcome!
Justin _______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
-- Jonathan Aquilina _______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
Another question if this is a single wiki why not again rsync that across the 4 servers that way when you update one you have them all updated easily via rsync when a new release of MW comes out.
I think its best I step back on this as I am no wiki expert at all. I can provide solutions to certain issues in terms of server layout and what not but that is about it :(
On Tue, Oct 28, 2014 at 7:14 AM, Justin Lloyd jclbugz@gmail.com wrote:
I think my explanation was not the clearest it could have been. Let's say for the moment that I have one wiki. That wiki is served by a load balancer in front of a server farm consisting of four Apache vhosts, one per physical server, each with its own copy of MediaWiki, LocalSettings.php, etc. Thus, a request for say http://wiki.domain.com/wiki/Main_Page (and thus all of its included images, css, js, etc.) is actually distributed by the load balancer across the four vhosts. Each of the four physical hosts NFS mounts the same shared directory from the single NFS server so that all four Apache vhosts have simultaneous read-write access to the same uploaded multimedia content.
In case that description is missing your point, I'll add that I do indeed rsync the NFS server's shared directory to another server nightly (I could easily shorten that interval), which in turn gets rsynced to an offsite server. So the NFS server is a single point of failure, but I do have both local and remote copies of the uploaded content. My desire is for increased reliability of that backend fileserver.
Does that answer your question or am I still missing your point? :)
Justin
On Mon, Oct 27, 2014 at 10:41 PM, Jonathan Aquilina < eagles051387@gmail.com> wrote:
You are mentioning NFS why not use rsync to replicate to a 2ndary nfs server and set it to run lets say every 5 to 10 min or how ever often you want to keep the 2ndary server updated.
On Tue, Oct 28, 2014 at 1:13 AM, Justin Lloyd jclbugz@gmail.com wrote:
Hi all,
Currently I have five wikis with the largest one being about 35k
articles
(109k pages) and pretty heavily trafficked. My basic server
architecture
is
four web servers behind a load balancer and with a single NFS server
that
shares out a directory that contains the upload directory content for
each
of the five wikis, e.g. /wiki/wiki1, /wiki/wiki2, etc. (There are also MySQL and Memcached servers but they are not relevant to this
discussion.)
Each web server mounts /wiki in one location, say /var/www/images and
each
of the five MediaWiki instances on the server has its images
subdirectory
as a symlink to its corresponding subdirectory under the mount, e.g. /var/www/images/wiki2.
Obviously the NFS server is a single point of failure but I've yet to
come
up with a good alternative shared-filesystem architecture that doesn't require an expensive license like SNFS.
Finally, I'm considering moving the whole shebang to AWS but using S3 directly on the web servers doesn't seem viable in this architecture.
So I'm wondering how others are approaching the design of load
balancing
(multiple instances of) MediaWiki across multiple web servers while maintaining a single source for each wikis upload directory content.
I'm
willing to COMPLETELY reevaluate my wiki server architecture as long as it's fast and highly available, so all suggestions are welcome!
Justin _______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
-- Jonathan Aquilina _______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
No problem, I'm glad to answer your questions to ensure I'm providing all relevant info. I do have five wikis, not just one as used in the previous explanation. Each of the five wikis has its own Apache vhost and documentroot directory on each of the four servers, making for 20 copies of MediaWiki. (I'm thinking of moving to a wiki family http://www.mediawiki.org/wiki/Manual:Wiki_family architecture to reduce that to one copy per server, but that will take a lot of work to refactor LocalSettings.php into separate files to split out common and wiki-specific configs that can be require()d, including Jinja templating for use by SaltStack states to generate the actual configs based on a centralized dataset in Salt Pillar.)
Each of the four servers has a single NFS mount and each of the 20 wiki directories has a symlink to its corresponding subdirectory in the NFS mount. For example, on each server the vhosts DocumentRoots are /var/www/sites/wiki1 through /var/www/sites/wiki5. The NFS-shared directory is mounted on each of the web servers as /var/www/images and each wiki DocumentRoot has a symlink called images to a subdirectory of that, e.g. /var/www/sites/wiki1/images is a symlink to /var/www/images/wiki1, which contains the hashed upload contents http://www.mediawiki.org/wiki/Manual:$wgHashedUploadDirectory.
And to directly address your question, even if this were a single wiki, there is plenty of traffic (multiple accesses per second per server) to prevent rsync from being viable for synchronization across the four servers since every page request results in multiple subrequests that get distributed across all four servers. (I may also have more than four servers in the future since I'll move to AWS and gain the potential to easily scale horizontally, possibly even dynamically, using CloudFormation and Elastic Load Balancing.) Basically all of my web servers need that simultaneous read-write access to the same content and with appropriate write-lock semantics such as that provided by NFS.
Again, perhaps there's a better way to architect a resilient set of wikis that would simplify this design, and I'm open to all suggestions, so far what I have is the best I've come up with in the time I've managed these wikis. The current architecture has had its pros and cons but has worked reasonbly well. I'm just reevaluating things in light of the move to AWS and taking a step back to see if I can improve the reliability and simplify the management of the environment.
Justin
On Mon, Oct 27, 2014 at 11:27 PM, Jonathan Aquilina eagles051387@gmail.com wrote:
Another question if this is a single wiki why not again rsync that across the 4 servers that way when you update one you have them all updated easily via rsync when a new release of MW comes out.
I think its best I step back on this as I am no wiki expert at all. I can provide solutions to certain issues in terms of server layout and what not but that is about it :(
On Tue, Oct 28, 2014 at 7:14 AM, Justin Lloyd jclbugz@gmail.com wrote:
I think my explanation was not the clearest it could have been. Let's say for the moment that I have one wiki. That wiki is served by a load
balancer
in front of a server farm consisting of four Apache vhosts, one per physical server, each with its own copy of MediaWiki, LocalSettings.php, etc. Thus, a request for say http://wiki.domain.com/wiki/Main_Page (and thus all of its included images, css, js, etc.) is actually distributed
by
the load balancer across the four vhosts. Each of the four physical hosts NFS mounts the same shared directory from the single NFS server so that
all
four Apache vhosts have simultaneous read-write access to the same
uploaded
multimedia content.
In case that description is missing your point, I'll add that I do indeed rsync the NFS server's shared directory to another server nightly (I
could
easily shorten that interval), which in turn gets rsynced to an offsite server. So the NFS server is a single point of failure, but I do have
both
local and remote copies of the uploaded content. My desire is for
increased
reliability of that backend fileserver.
Does that answer your question or am I still missing your point? :)
Justin
On Mon, Oct 27, 2014 at 10:41 PM, Jonathan Aquilina < eagles051387@gmail.com> wrote:
You are mentioning NFS why not use rsync to replicate to a 2ndary nfs server and set it to run lets say every 5 to 10 min or how ever often
you
want to keep the 2ndary server updated.
On Tue, Oct 28, 2014 at 1:13 AM, Justin Lloyd jclbugz@gmail.com
wrote:
Hi all,
Currently I have five wikis with the largest one being about 35k
articles
(109k pages) and pretty heavily trafficked. My basic server
architecture
is
four web servers behind a load balancer and with a single NFS server
that
shares out a directory that contains the upload directory content for
each
of the five wikis, e.g. /wiki/wiki1, /wiki/wiki2, etc. (There are
also
MySQL and Memcached servers but they are not relevant to this
discussion.)
Each web server mounts /wiki in one location, say /var/www/images and
each
of the five MediaWiki instances on the server has its images
subdirectory
as a symlink to its corresponding subdirectory under the mount, e.g. /var/www/images/wiki2.
Obviously the NFS server is a single point of failure but I've yet to
come
up with a good alternative shared-filesystem architecture that
doesn't
require an expensive license like SNFS.
Finally, I'm considering moving the whole shebang to AWS but using S3 directly on the web servers doesn't seem viable in this architecture.
So I'm wondering how others are approaching the design of load
balancing
(multiple instances of) MediaWiki across multiple web servers while maintaining a single source for each wikis upload directory content.
I'm
willing to COMPLETELY reevaluate my wiki server architecture as long
as
it's fast and highly available, so all suggestions are welcome!
Justin _______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
-- Jonathan Aquilina _______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
-- Jonathan Aquilina _______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
On Oct 28, 2014 2:56 AM, "Justin Lloyd" jclbugz@gmail.com wrote:
Again, perhaps there's a better way to architect a resilient set of wikis that would simplify this design, and I'm open to all suggestions, so far what I have is the best I've come up with in the time I've managed these wikis.
You should take a look at how WMF handles this. The wiki farm stuff (aka hetdeploy/mwmultiversion) is documented at https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment (maybe up to date because edited recently)
also: * https://github.com/wikimedia/operations-mediawiki-config/blob/master/multive... * https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-con... * https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-con...
(and see other files in that dir too)
WMF essentially runs its own s3. (called swift. An openstack project) I don't see why you couldn't use s3 in a similar way. might be overkill for you to run your own swift cluster. Rackspace has a public swift cluster you could use too.
It doesn't look like there's currently an s3 filebackend but you're welcome to add one.
* https://github.com/wikimedia/mediawiki/tree/master/includes/filebackend * https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-con... * https://wikitech.wikimedia.org/wiki/Media_storage * our InitialiseSettings.php: wgUploadPath
In the short term you could load balance reads with a more frequent rsync and then it's only spof for writes and rendering file description pages. I think. (wgUploadPath points to load balancer for a cluster of machines that all have a local copy of all images.)
-Jeremy
Hmm, interesting. This will take time for me to digest and I need to get to bed. However, I will say that adding an S3 backend is beyond me since I'm a sysadmin, not a developer, and I haven't done any significant PHP work in about 12 years.
Also, unless I'm missing something or being dense (it is late here), rsync simply wouldn't work since the upload directories are constantly being accessed and files written through one web server could easily be immediately accessed afterwards through another web server, and since there are four web servers (and possibly more or even less if I were to add AWS Auto Scaling into the mix), so keeping them all identical when writes could go through any of them would be pretty much impossible.
Also, we've seen problems in the past with the master-write / replica-read structure due to issues with MySQL 5.5 (since we're running Ubuntu 12.04 currently), such as temp tables getting created on the replica but then writes to those tables would fail due to the replica being marked read-only, so for now we have all reads and writes going to the master with the replica just being there for backups (via cron jobs running Percona's Innobackupex several times a day). We'd likely move from our current physical MySQL servers to AWS RDS, so perhaps that would eliminate the past issues we've encountered...
Thanks for the links and suggestions. I'll investigate them tomorrow.
Justin
On Tue, Oct 28, 2014 at 1:01 AM, Jeremy Baron jeremy@tuxmachine.com wrote:
On Oct 28, 2014 2:56 AM, "Justin Lloyd" jclbugz@gmail.com wrote:
Again, perhaps there's a better way to architect a resilient set of wikis that would simplify this design, and I'm open to all suggestions, so far what I have is the best I've come up with in the time I've managed these wikis.
You should take a look at how WMF handles this. The wiki farm stuff (aka hetdeploy/mwmultiversion) is documented at https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment (maybe up to date because edited recently)
also:
https://github.com/wikimedia/operations-mediawiki-config/blob/master/multive...
https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-con...
https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-con...
(and see other files in that dir too)
WMF essentially runs its own s3. (called swift. An openstack project) I don't see why you couldn't use s3 in a similar way. might be overkill for you to run your own swift cluster. Rackspace has a public swift cluster you could use too.
It doesn't look like there's currently an s3 filebackend but you're welcome to add one.
https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-con...
- https://wikitech.wikimedia.org/wiki/Media_storage
- our InitialiseSettings.php: wgUploadPath
In the short term you could load balance reads with a more frequent rsync and then it's only spof for writes and rendering file description pages. I think. (wgUploadPath points to load balancer for a cluster of machines that all have a local copy of all images.)
-Jeremy _______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
On Tue, Oct 28, 2014 at 8:33 AM, Justin Lloyd jclbugz@gmail.com wrote:
Also, unless I'm missing something or being dense (it is late here), rsync simply wouldn't work since the upload directories are constantly being accessed and files written through one web server could easily be immediately accessed afterwards through another web server, and since there are four web servers (and possibly more or even less if I were to add AWS Auto Scaling into the mix), so keeping them all identical when writes could go through any of them would be pretty much impossible.
Well you don't have to limit yourself to having the file uploads visible in only one part of the filesystem. (so this could work even if you don't have dedicated storage hosts. mediawiki uses NFS mount and unrelated vhost serves static files out of rsync target)
But you could also do like WMF (and it sounds like you already have dedicated storage boxes?): files are fetched from one hostname/varnish cluster/storage cluster and HTML/etc. comes from a completely separate hostname/varnish cluster/php cluster.
4 webservers mount rsync master by NFS. same as now. writes and file description page rendering runs over NFS.
new webservers (or vhosts on existing webservers or webservers on the storage hosts directly) serve images read-only from the local copy of the files propagated by the rsync cron. (or whatever other way)
you could autoscale for the 4 webservers that don't have local images at all (just NFS) and then either build an initial rsync into the scale up process for storage hosts or do that scaling manually.
anyway, this is all just to address the immediate spof quickly. longer term maybe figure out a way to use s3 or something. (which you could do already actually. your rsync cron could instead be a copy to s3 cron. but then still spof on the same things where the config described above would also have spof. e.g. file description pages, uploading, deleting, etc.)
-Jeremy
Jeremy,
The wiki family files to which you linked are very interesting and if I go the route of a wiki family, I'd do a very scaled down and simpler version of that, as I only have five very similar wikis (they're essentially just different language versions of the same thing, though their content is managed by different groups of editors, and not by me). I'd seen the files at https://noc.wikimedia.org/conf/ previously, so I had a rough idea of that approach.
As for your suggestions about the uploaded files, I should first reiterate that currently I have a single storage host and that is my primary concern. Using rsync for backups is fine, but not to keep multiple production storage hosts synchronized. Based on some more reading today, I think it would take moving to a clustered filesystem such as GFS2 http://en.wikipedia.org/wiki/GFS2. I am curious though about the server architecture and what the MediaWiki configuration would be to use the file servers and image scalers shown in this diagram http://upload.wikimedia.org/wikipedia/commons/d/d8/Wikimedia-servers-2010-12-28.svg .
Justin
On Tue, Oct 28, 2014 at 6:19 AM, Jeremy Baron jeremy@tuxmachine.com wrote:
On Tue, Oct 28, 2014 at 8:33 AM, Justin Lloyd jclbugz@gmail.com wrote:
Also, unless I'm missing something or being dense (it is late here),
rsync
simply wouldn't work since the upload directories are constantly being accessed and files written through one web server could easily be immediately accessed afterwards through another web server, and since
there
are four web servers (and possibly more or even less if I were to add AWS Auto Scaling into the mix), so keeping them all identical when writes
could
go through any of them would be pretty much impossible.
Well you don't have to limit yourself to having the file uploads visible in only one part of the filesystem. (so this could work even if you don't have dedicated storage hosts. mediawiki uses NFS mount and unrelated vhost serves static files out of rsync target)
But you could also do like WMF (and it sounds like you already have dedicated storage boxes?): files are fetched from one hostname/varnish cluster/storage cluster and HTML/etc. comes from a completely separate hostname/varnish cluster/php cluster.
4 webservers mount rsync master by NFS. same as now. writes and file description page rendering runs over NFS.
new webservers (or vhosts on existing webservers or webservers on the storage hosts directly) serve images read-only from the local copy of the files propagated by the rsync cron. (or whatever other way)
you could autoscale for the 4 webservers that don't have local images at all (just NFS) and then either build an initial rsync into the scale up process for storage hosts or do that scaling manually.
anyway, this is all just to address the immediate spof quickly. longer term maybe figure out a way to use s3 or something. (which you could do already actually. your rsync cron could instead be a copy to s3 cron. but then still spof on the same things where the config described above would also have spof. e.g. file description pages, uploading, deleting, etc.)
-Jeremy
MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
mediawiki-l@lists.wikimedia.org