The problem: In general,
rationalwiki.org is melting under the strain.
We can't quite afford the next step up right this moment, though want
to plan for it, and need to keep things from intermittently blowing up
as they are.
We live on a 4GB Linode. I don't have a full handle on what we've got
on it - it's accreted in an ad-hoc fashion with some horrible bodges -
but there's several MediaWikis, only one of which (
rationalwiki.org)
has any appreciable traffic. There's Lucene search, which is of course
huge in memory.
rationalwiki.org is running via libphp5, the other
wikis are via fastcgi (for no particular reason). The wikis have a
huge pile of extensions, e.g.
http://rationalwiki.org/wiki/Special:Version , a lot of which are
local things.
Whenever an article hits Reddit,[1] the server suffers under the load.
Typically it goes into swap and thrashes itself to death. If we're
really lucky the oom-killer comes out to play and shoots things
randomly (usually Apache, maybe Lucene). The fun bit: sometimes it
does this for no visible reason, just tips over into swap and promptly
stops talking to the world (my shell session still works slowly).
Per advice from here I've re-enabled keepalive, setting it to 10
requests and 2 seconds. This does speed things up a bit for the users.
I haven't put a cache on the front as yet, still letting MediaWiki's
file cache do that. apc is running (99.8% cache hit rate, so that's
fine) and memcached (90% cache hit rate, upped it from 64MB to 128MB
and it's still about 90%).
So second thing we need to work out what our next stage of evolution
is (the 8GB Linode for $200/mo more? [2] or multiple servers?
Distributed how?) - and the first thing we need to work out how not to
make the box melt in the meantime.
Clues welcomed!
- d.
[1] usually a joke article, like
http://rationalwiki.org/wiki/Scientific_evidence_of_evolution_being_a_hoax
... nothing we've actually worked hard on *hand to forehead*
[2] this appears pretty typical pricing for non-crappy hosting