Some thoughts: * Use a front-end web cache. The old 'file cache' in MediaWiki probably doesn't work as well as it once did, especially since you're on a recent version of MediaWiki that uses ResourceLoader -- that'll mean dynamic hits happen for styles and scripts even on a cached page -- and even that cached page is going through MediaWiki and requires CPU time and memory. I'm not sure whether any of the cloud-based CDNs play nicely with MediaWiki these days, but you can certainly set up Squid or Varnish on a second Linode box. A proper caching setup that covers both article views and ResourceLoader (load.php stuff) should be able to avoid a *lot* of hits to PHP as long as people are just clicking around. You can also do things like loading static resources (uploaded files, skin images etc) from a separate server, but putting a cache in front should cover much the same requirements as that. * Consider splitting Lucene search to a separate server. Particularly since it's memory intensive, splitting it out will leave more RAM on your main server for handling spikes. * Do some load tests and set maximums appropriately If you've got a lot of MediaWiki hits going simultaneously, that's going to drive up your memory usage. Make sure that hitting the maximum number of server threads doesn't kill you just from loading up Apache+PHP+MediaWiki! -- brion On Wed, Oct 17, 2012 at 3:25 PM, David Gerard <dgerard(a)gmail.com> wrote:

On Thu, October 18, 2012 00:25, David Gerard wrote: Have you taken a good look at your database configuration? Are you using InnoDB tables, have you allocated enough memory and threads, etc? http://hackmysql.com/mysqlreport (assuming you're running MySQL) may provide some insight. Siebrand

A few things we've learned over the years through trial and error that may be helpful to you. I think your biggest improvement is probably getting a Squid/Varnish cache setup (as recently discussed here). Depending on your traffic (logged in vs anonymous viewers) you could see as much as a x6 drop in Apache requests. It also greatly helps with the Slashdot/Reddit effect as most of the views in such a traffic spike will be anonymous viewers which would be handled entirely by Squid/Varnish. It is possible to squeeze Squid into the same server but you have to juggle ports and memory. You may also be able to reduce the Apache load significantly by moving all static resources (images, CSS, JS) to be served by a lighter weight server like lighttpd or nginx. I guess recently some CSS/JS was changed from static files to the ResourceLoader but there is still plenty of static content to consider it (our lighttpd server averages x10 more connections than our Apache server). Like Squid you can squeeze this into the same server by juggling ports and will probably actually save memory (a lighttpd/nginx connection uses far, far less memory than Apache would). The net result of these two changes is to try and minimize Apache connections, which are relatively CPU/memory expensive, for only what is needed. As a point of comparison we're currently averaging 170 req/sec over all servers (Squid + lighttpd + Apache) but Apache is only handling 10 req/sec. We simply couldn't handle the traffic if we tried to get Apache to handle all of it. One quick setting you can look into is to set $wgCacheDirectory to a local directory. We had huge problems recently upgrading from 1.14 to 1.19 until we set this parameter. It may have just been related to our upgrade but anything that reduces database usage, especially on a single server, is usually a good thing. The only other significant advice I can think of is to make sure you regularly benchmark/load test as you try to optimize things. There are a bunch of good reasons to do this: 1) Documentation 2) Verification : i.e., did we actually make things faster and if so by how much 3) Load Capacity Planning: how many connections/users can we support and how close are we right now to the limit I usually just use ApacheBench (ab) which gives good enough results but there are plenty of others and you can get as complex testing as you want. As for hardware, we use iWeb which has reasonable prices although its support it hit or miss (its either great or terrible it seems). We have a variety of dedicated servers in the $100/month range having gone for a horizontal scaling design (more smaller boxes with some redundancies among them). I don't know what resources (money) you have available and what sort of request rate you you're getting but with some optimization and 1 or 2 boxes I would think you should be able to survive a Slashdot/Reddit effect without completely going under. Beyond this your next step is either scaling up (faster computer) or out (more servers). For scaling out just look at your process usage (top) and see what is using the most CPU/memory and then split that out to the new server. Chances are a good first step would be getting a dedicated database server or another Apache server. On 17 October 2012 18:25, David Gerard <dgerard(a)gmail.com> wrote: -- Dave Humphrey -- dave(a)uesp.net Founder/Server Admin of the Unofficial Elder Scrolls Pages -- www.uesp.net

We did the following on our MediaWiki server with 3000 users and 150,000 articles: - Moved Lucene to a separate server. HUGE HUGE benefit in lowering memory use (a - Installed Extension:APC. 50% speedup in page rendering. - Installed memcached. Another 50% speedup in page rendering. - Code-reviewed our custom-written extensions and optimized them. Another 50% speedup in page rendering. We also increased physical RAM to 6GB (from 2GB), which may or may not be an option for you. DanB