So,
rationalwiki.org has been *much* faster and more usable with a
coupla squids and a load-balancer in front of the
Apache/Lucene/database node. (We could probably cope with just one
squid, but Trent wanted to experiment.) The nodes are all Ubuntu 10.04
Linodes, the software manually kept up to date.
Our problem now is that the Apache box sometimes ... just goes nuts,
fills memory with Apache processes, then it goes into swap, then the
oom-killer comes out to play and we have to work out what it's killed
or (quicker) reboot the node.
We have had occasional load spikes - where the load-balancer sees
someone or something hammering it at 300 hits/sec or so - but they
*don't* always coincide with Apache going nuts. The squids don't show
any excess load during Apache going nuts either.
If we happen to catch it when it's in swap but before oom-killer comes
out, apache2ctl restart brings things back to normality.
The Apache node has 4GB memory, about 3GB of that being free/cache in
normal operation.
We have NO IDEA what or why this is happening. Last happened around
three days ago. Since then it's been lovely, but it always is until it
falls over. Clues welcomed.
- d.