Hi again,
_in short_:
* did an installation of MediaWiki 1.9.x a few weeks ago; * MediaWiki was working fine as expected and as used for several weeks; * MediaWiki suddenly stopped working partly and became mostly inaccessible a feew days ago; * Currently, it started to interfere with other services runnin on the same host (requesting a page from Media Wiki kills apache2) * can't find any helpful information in ../syslog, ../messages, the apache2's error.log or access.log; enabled php.log (in php.ini) and debug.log (in LocalSettings.php): the first logs nothing helpful, also, the latter is not being created at all.
_in full_:
A few months ago, I installed MediaWiki 1.9, created some hundred nodes and all appeared to work like a charm: Creating articles, deleting, moving, or editing them, all reasonably fast and relaible.
Now, after several weeks of operation, the installation has partially ceased to work: Entering an existing term in the Search box has no effect - the browser claims to request the page, but nothing happens (nothing = no page showing up, and no timeout appearing after several minutes [hours]). Sometimes - but not reproducable - the requested page appears, when I directly enter the page's full URL in the browser's location bar. Creating new pages seems no longer to work at all. I haven't changed the configuration in LocalSettings.php for weeks, and I haven't noticed having changed anything else on the server (running under Debian "Etch" since it was released).
However, when I do a "tail -f" on the webserver's logfiles, requests from spiders and external browsers are hushing by with a pretty high frequency (several files requested per second). What appears *really* strange to me is, that requests for pages from my own browser (Opera 9.2.1/Win) don't appear in access.log, as long as I use Opera, where I'm logged in to the Wiki; when I switch to Firefox (2.0.0.4/Win, not logged in to the Wiki), those requests start to appear in access.log.
Sticking with Firefox, entering page names of existing articles results in delivering the article and showing up in access.log; however, when entering non-existing article names (in order to create them), those requests don't appear in access.log, and MediaWiki delivers no page (search result etc.), like with Opera. It's also mostly the same result, if I enter a new term in the Search box or directly via the URL location (.../w/index.php?title=New_Article&action=edit). Directly after rebooting the server, sometimes creating new articles does work again when directly entering the new article's URL, but only once.
Another LAMP application on the same host (Drupal) is working fine (more or less, and very slow, but however it's still working), so basically Apache, MySQL, and PHP seem to work.
The server hosts three domains, for each one exists one MediaWiki installation; two of them were set up completely and checked for correct operation a few weeks ago, but they were not in use yet. These installations stopped working also and show similar behaviour to those described below.
Somehow requests to MediaWiki started a few days ago to interfere with the other services operation; e.g., when I'm requesting a page from MediaWiki (e.g. Front Page) and concurrently try to save an article in Drupal, the server does neither deliver the FrontPage from MediaWiki, nor save the article in Drupal (I'm waiting for approx. 10 minutes and usually cancel then the operation by pressing "Esc" in the browser). At this moment, all apache2 proecess dissapear from top/htop and no other requests to either Drupal or MediaWiki are served in any way. As it seeems, MediaWiki started for an unknown reason to kill apache2 in the setup im running.
Drupal starts responding again as soon as I do an "/etc/init.d/apache2 restart". Sometimes even this fails:
# /etc/init.d/apache2 restart Forcing reload of web server (apache2)...(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80 no listening sockets available, shutting down Unable to open logs
Apache2 seems to die completely:
# /etc/init.d/apache2 stop Stopping web server (apache2)...httpd (no pid file) not running
It then can be restarted;
# /etc/init.d/apache2 start Starting web server (apache2)....
After this, normal operation continues, as long as I don't request any page from MediaWiki. If I request the FrontPage of any of those three setupts, this repeats all over again.
I tried to check the MediaWiki databases with phpMyAdmin (2.9.1.1-Debian-3); even opening them takes over two minutes, checking single small tables like "de_watchlist" results in an "OK", checking large tables like "de_text" or even all at once does not finish after several hours (also without phpMyAdmin giving a timeout). During this "check" operations, CPU load according to "htop" fluctuates between 16 and 97% (most of the time it's around or below 50%)).
This is a pretty bizarre scenery: In the background of my desktop is a shell, where requests logged to /var/log/apache2/access.log continuously are floating by, on the same time being unable to load or even edit a single page on my own Wiki... ;-/
/var/log/syslog and ../messages don't show anything unusual.
In LocalSetting.php, I enabled
$wgShowExceptionDetails = true; $wgDebugLogFile = "debug.log";
However, a "debug.log" is not being created in the wiki directory (or according to "updatedb; locate" anywhere else on the server).
Also, the php error log is pretty quiet; after doing an "apache2 restart", it reports once:
-- snip --
# tail -f php.log [29-Jun-2007 01:39:12] PHP Warning: Module 'gd' already loaded in Unknown on line 0 [29-Jun-2007 01:39:12] PHP Warning: Module 'mysql' already loaded in Unknown on line 0
-- snip --
When entering some expression in the search box, php.log logs nothing else (at least not for several minutes). While waiting for something to happen, requests from external hosts are rushing by, also not causing php errors.
/etc/php5/apache2/php.ini (I hope that's the right one) is set to:
... error_reporting = E_ALL & ~E_NOTICE display_errors = On display_startup_errors = Off log_errors = On ignore_repeated_source = Off report_memleaks = On ...
Any hints where and how I could look what's going wrong?
Thanks & regards, -asb