A review of the uptime results suggests that we aren't quite as fast as we were in the previous version (when I essentially disabled getOtherNamespaces), but we're doing o.k. now. I think we still have some work to do.

Axel Boldt wrote:

I have been wondering before whether it's possible to have memory leaks in php scripts: is everything automatically released once the page is served?

We are running apc php caching: http://apc.communityconnect.com/

In practice this means that all the .php scripts get precompiled and saved to disk as .php_apc. They are precompiled and cached so that "the overhead of parsing and compiling can almost completely be eliminated". This might also introduce a new place for memory leaks to happen.

However, the machine does not appear to be swapping most of the time. I suspect that when the load really spikes up (review http://www.wikipedia.com/uptime.txt for a log of recent uptime output), we are swapping.

Somebody somewhere could be eating memory, and that would slow everything. If memory is indeed the bottleneck, then caching might not be a good idea, since we in effect transport twice as much data between database and php script, and I would expect that the data is buffered in memory on both ends.

Possibly.

Some top/ps outputs would really help.

This is a typical looking 'top'.

8:54am up 42 days, 23:24, 1 user, load average: 6.05, 4.93, 5.15 172 processes: 163 sleeping, 8 running, 1 zombie, 0 stopped CPU states: 73.3% user, 26.2% system, 0.0% nice, 0.3% idle Mem: 1029824K av, 934008K used, 95816K free, 0K shrd, 49900K buff Swap: 506008K av, 64200K used, 441808K free 517092K cached

PID USER PRI NI SIZE RSS SHARE STAT LIB %CPU %MEM TIME COMMAND 7887 root 20 0 51632 45M 28972 R 0 27.5 4.4 0:04 mysqld 12467 nobody 19 0 8516 7856 5076 R 0 4.3 0.7 0:03 httpd 8154 nobody 20 0 8868 8104 5640 R 0 3.8 0.7 0:04 httpd 12468 root 12 0 51632 45M 28972 S 0 3.8 4.4 0:01 mysqld 17146 nobody 19 0 7724 7068 4128 R 0 3.8 0.6 0:00 httpd 17141 jwales 19 0 936 936 672 R 0 3.3 0.0 0:00 top 16989 nobody 14 0 8280 7712 4900 R 0 3.0 0.7 0:01 httpd 17181 nobody 14 0 3424 3424 956 R 0 3.0 0.3 0:00 wiki.cgi 17136 root 10 0 51632 45M 28972 S 0 2.3 4.4 0:00 mysqld 7879 nobody 9 0 8772 8116 4728 S 0 2.1 0.7 0:03 httpd 17182 nobody 14 0 1896 1896 896 R 0 2.0 0.1 0:00 q.cgi 7723 nobody 17 0 7820 7096 4676 S 0 1.8 0.6 0:03 httpd 17148 nobody 11 0 7884 7256 4452 S 0 1.8 0.7 0:00 httpd 17170 nobody 9 0 7108 6480 4096 S 0 1.6 0.6 0:00 httpd 17158 nobody 9 0 7780 7152 4452 S 0 1.5 0.6 0:00 httpd 17165 nobody 10 0 7000 6372 4064 S 0 1.5 0.6 0:00 httpd 9949 nobody 11 0 0 0 0 Z 0 1.3 0.0 0:00 httpd <defunct> 17144 nobody 10 0 7876 7248 4452 S 0 1.3 0.7 0:00 httpd 17134 nobody 10 0 7560 6936 4260 S 0 1.1 0.6 0:00 httpd 17058 nobody 10 0 7660 7048 4384 S 0 1.0 0.6 0:00 httpd 17067 nobody 10 0 8032 7420 4288 S 0 1.0 0.7 0:00 httpd 17143 nobody 12 0 7020 6432 4120 S 0 1.0 0.6 0:00 httpd 17159 nobody 11 0 7080 6452 4132 S 0 1.0 0.6 0:00 httpd 17166 nobody 9 0 7068 6440 4128 S 0 1.0 0.6 0:00 httpd 17169 nobody 9 0 7696 7084 4468 S 0 1.0 0.6 0:00 httpd 16958 nobody 10 0 8024 7444 4632 S 0 0.8 0.7 0:03 httpd 16978 nobody 11 0 7976 7352 4696 S 0 0.8 0.7 0:01 httpd 16983 nobody 10 0 7960 7348 4696 S 0 0.8 0.7 0:02 httpd 17084 root 9 0 51632 45M 28972 S 0 0.8 4.4 0:05 mysqld 17167 nobody 9 0 7112 6484 4148 S 0 0.8 0.6 0:00 httpd 17168 nobody 9 0 6988 6360 4068 S 0 0.8 0.6 0:00 httpd 17083 nobody 10 0 7336 6724 4228 S 0 0.6 0.6 0:00 httpd 7730 nobody 10 0 8632 7884 5444 S 0 0.5 0.7 0:04 httpd 7850 nobody 10 0 8556 7836 5444 S 0 0.5 0.7 0:03 httpd 16985 nobody 10 0 7236 6648 4240 S 0 0.5 0.6 0:00 httpd 17052 nobody 9 0 8144 7572 4668 S 0 0.5 0.7 0:00 httpd 17063 root 9 0 51632 45M 28972 S 0 0.5 4.4 0:05 mysqld 17145 nobody 9 0 7028 6404 4072 S 0 0.5 0.6 0:00 httpd 7851 root 10 0 51632 45M 28972 S 0 0.3 4.4 0:00 mysqld