I suspect the problem is somewhere in MediaWiki. If knew where I'd be posting it here.
- I've stripped away other things on the server. Pretty much the only thing left is the most recent version of MediaWiki (1.23.3) and Debian (stable) packages. - I ran 'debsums' to check whether any of the Debian packages are corrupt. - I am not aware of any LAMP bugs in Debian.
The MediaWiki install is near fresh. I did a manual upgrade for version 1.23.1 and then used the patches to get to 1.23.3. The upgrade from 1.23.1 to 1.23.2 had a bug in it-- but I think that was inconsequential. In any case, that upgrade was done July 30, several weeks preceding the problems.
Here is a snippet from 'ps aux': ----- USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND ... www-data 10518 0.0 2.8 291376 57052 ? S 20:39 0:03 /usr/sbin/apache2 -k start www-data 10567 0.0 2.5 288076 51480 ? S 20:42 0:02 /usr/sbin/apache2 -k start www-data 10568 0.0 2.6 290496 54140 ? S 20:42 0:04 /usr/sbin/apache2 -k start www-data 10569 0.0 2.5 288820 51840 ? S 20:42 0:02 /usr/sbin/apache2 -k start www-data 10570 0.0 2.6 291384 53036 ? S 20:42 0:02 /usr/sbin/apache2 -k start www-data 10571 0.0 2.9 293316 58812 ? S 20:42 0:05 /usr/sbin/apache2 -k start www-data 10572 0.0 2.7 290976 55360 ? S 20:42 0:04 /usr/sbin/apache2 -k start www-data 10573 0.0 2.7 290888 55820 ? S 20:42 0:05 /usr/sbin/apache2 -k start www-data 10649 0.0 2.7 290728 54860 ? S 20:53 0:05 /usr/sbin/apache2 -k start www-data 10650 0.0 2.7 293000 55924 ? S 20:53 0:04 /usr/sbin/apache2 -k start ... ----- The memory seems to be going into apache2.
Earlier this morning I rebooted the server. About three hours later the images were gone again. After rebooting this morning... the server has been working okay (so far about 13 hours). I don't understand the cycle: why it breaks in 3 hours... or goes for more than 12.
Some ideas I have: 1. Running a 'diff' of my active server against the tarball for the most recent MediaWiki-- to look for corrupted files.
2. At this point I am thinking about the extensions and doing a fresh install of MediaWiki... with the hope that will resolve things.
3. I have bimonthly images going back a year-- I can dump the database, rollback and grab the latest database changes.
My inclination is to try #2. It isn't that hard to do... if it works I can be pretty sure it was MediaWiki... and it solves the problem I have at the moment.
Any thoughts/ideas are welcome.
Michael