I think my explanation was not the clearest it could have been. Let's say for the moment that I have one wiki. That wiki is served by a load balancer in front of a server farm consisting of four Apache vhosts, one per physical server, each with its own copy of MediaWiki, LocalSettings.php, etc. Thus, a request for say http://wiki.domain.com/wiki/Main_Page (and thus all of its included images, css, js, etc.) is actually distributed by the load balancer across the four vhosts. Each of the four physical hosts NFS mounts the same shared directory from the single NFS server so that all four Apache vhosts have simultaneous read-write access to the same uploaded multimedia content. In case that description is missing your point, I'll add that I do indeed rsync the NFS server's shared directory to another server nightly (I could easily shorten that interval), which in turn gets rsynced to an offsite server. So the NFS server is a single point of failure, but I do have both local and remote copies of the uploaded content. My desire is for increased reliability of that backend fileserver. Does that answer your question or am I still missing your point? :) Justin On Mon, Oct 27, 2014 at 10:41 PM, Jonathan Aquilina <eagles051387(a)gmail.com> wrote:

No problem, I'm glad to answer your questions to ensure I'm providing all relevant info. I do have five wikis, not just one as used in the previous explanation. Each of the five wikis has its own Apache vhost and documentroot directory on each of the four servers, making for 20 copies of MediaWiki. (I'm thinking of moving to a wiki family <http://www.mediawiki.org/wiki/Manual:Wiki_family> architecture to reduce that to one copy per server, but that will take a lot of work to refactor LocalSettings.php into separate files to split out common and wiki-specific configs that can be require()d, including Jinja templating for use by SaltStack states to generate the actual configs based on a centralized dataset in Salt Pillar.) Each of the four servers has a single NFS mount and each of the 20 wiki directories has a symlink to its corresponding subdirectory in the NFS mount. For example, on each server the vhosts DocumentRoots are /var/www/sites/wiki1 through /var/www/sites/wiki5. The NFS-shared directory is mounted on each of the web servers as /var/www/images and each wiki DocumentRoot has a symlink called images to a subdirectory of that, e.g. /var/www/sites/wiki1/images is a symlink to /var/www/images/wiki1, which contains the hashed upload contents <http://www.mediawiki.org/wiki/Manual:$wgHashedUploadDirectory>. And to directly address your question, even if this were a single wiki, there is plenty of traffic (multiple accesses per second per server) to prevent rsync from being viable for synchronization across the four servers since every page request results in multiple subrequests that get distributed across all four servers. (I may also have more than four servers in the future since I'll move to AWS and gain the potential to easily scale horizontally, possibly even dynamically, using CloudFormation and Elastic Load Balancing.) Basically all of my web servers need that simultaneous read-write access to the same content and with appropriate write-lock semantics such as that provided by NFS. Again, perhaps there's a better way to architect a resilient set of wikis that would simplify this design, and I'm open to all suggestions, so far what I have is the best I've come up with in the time I've managed these wikis. The current architecture has had its pros and cons but has worked reasonbly well. I'm just reevaluating things in light of the move to AWS and taking a step back to see if I can improve the reliability and simplify the management of the environment. Justin On Mon, Oct 27, 2014 at 11:27 PM, Jonathan Aquilina &lt;eagles051387(a)gmail.com&gt; wrote:

Hmm, interesting. This will take time for me to digest and I need to get to bed. However, I will say that adding an S3 backend is beyond me since I'm a sysadmin, not a developer, and I haven't done any significant PHP work in about 12 years. Also, unless I'm missing something or being dense (it is late here), rsync simply wouldn't work since the upload directories are constantly being accessed and files written through one web server could easily be immediately accessed afterwards through another web server, and since there are four web servers (and possibly more or even less if I were to add AWS Auto Scaling into the mix), so keeping them all identical when writes could go through any of them would be pretty much impossible. Also, we've seen problems in the past with the master-write / replica-read structure due to issues with MySQL 5.5 (since we're running Ubuntu 12.04 currently), such as temp tables getting created on the replica but then writes to those tables would fail due to the replica being marked read-only, so for now we have all reads and writes going to the master with the replica just being there for backups (via cron jobs running Percona's Innobackupex several times a day). We'd likely move from our current physical MySQL servers to AWS RDS, so perhaps that would eliminate the past issues we've encountered... Thanks for the links and suggestions. I'll investigate them tomorrow. Justin On Tue, Oct 28, 2014 at 1:01 AM, Jeremy Baron &lt;jeremy(a)tuxmachine.com&gt; wrote:

Jeremy, The wiki family files to which you linked are very interesting and if I go the route of a wiki family, I'd do a very scaled down and simpler version of that, as I only have five very similar wikis (they're essentially just different language versions of the same thing, though their content is managed by different groups of editors, and not by me). I'd seen the files at https://noc.wikimedia.org/conf/ previously, so I had a rough idea of that approach. As for your suggestions about the uploaded files, I should first reiterate that currently I have a single storage host and that is my primary concern. Using rsync for backups is fine, but not to keep multiple production storage hosts synchronized. Based on some more reading today, I think it would take moving to a clustered filesystem such as GFS2 <http://en.wikipedia.org/wiki/GFS2>. I am curious though about the server architecture and what the MediaWiki configuration would be to use the file servers and image scalers shown in this diagram <http://upload.wikimedia.org/wikipedia/commons/d/d8/Wikimedia-servers-2010-12-28.svg> . Justin On Tue, Oct 28, 2014 at 6:19 AM, Jeremy Baron &lt;jeremy(a)tuxmachine.com&gt; wrote: