Hi,
I'm running Mediawiki 1.19.1 on Debian/stable server via lighttpd/php5-cgi. Earlier today I decided I wanted to enable wgDebugToolbar in my LocalSettings.php - done that, but now the wiki was not responding any more.
I've immediately reverted that change (and it really was only that change, I compared with the backups) but still the wiki was not responding.
top(1) showed that one of the php5-cgi process was using up a lot of CPU time, so I decided to kill that process and restarted the webserver. But still, accesssing the wiki was not possible: the page was "Loading...", but nothing got delivered back to the browser. Deleted cookies, tried another browser, still the wiki appeared to be unusable.
Looking at the MySQL database serving the wiki, I could see what was going on:
------ 307 Query REPLACE /* MessageBlobStore::updateModule 10.0.0.123 */ INTO `mw_msg_resource` (mr_resource,mr_lang,mr_blob,mr_timestamp) VALUES ('user.tokens','en','{}','20120926011737')
307 Query SELECT /* MessageBlobStore::updateModule 10.0.0.123 */ mr_blob FROM `mw_msg_resource` WHERE mr_resource = 'user.options' AND mr_lang = 'en' LIMIT 1 ------
These two queries where sent over and over again to the database by the php5-cgi process, ~10 times per second or so. Restarting the database did not help, as soon as I accessed the wiki again, these queries appeared again.
I've had this issues before, with 1.19.0, IIRC. I did not dig deeper though, somehow the next morning the wiki was "just" working again. But the issue was the same: changed something in the config and then these strange queries appeared.
I've put the LocalSettings.php and the database query log here:
https://trent.utfs.org/bits/mediawiki/
Some more notes:
* I've switched from lighttpd/php5-cgi to apache2/mod_php5, still the same: when the wiki is accessed, those strange queries appear and the wiki page "loads" until php's maximum execution time is exceeded..
* I'm using the APC cache. Disabling that (and wgMainCacheType) did no help either.
* I'm not using any Mediawiki extensions.
Anyone got an idea what might be going on here?
Thanks, Christian.
Thank you!
Rgds. Hartmut
-----Ursprüngliche Nachricht----- Von: mediawiki-l-bounces@lists.wikimedia.org [mailto:mediawiki-l-bounces@lists.wikimedia.org] Im Auftrag von Christian Kujau Gesendet: Mittwoch, 26. September 2012 09:55 An: mediawiki-l@lists.wikimedia.org Betreff: [MediaWiki-l] mediawiki drives the database insane
Hi,
I'm running Mediawiki 1.19.1 on Debian/stable server via lighttpd/php5-cgi. Earlier today I decided I wanted to enable wgDebugToolbar in my LocalSettings.php - done that, but now the wiki was not responding any more.
I've immediately reverted that change (and it really was only that change, I compared with the backups) but still the wiki was not responding.
top(1) showed that one of the php5-cgi process was using up a lot of CPU time, so I decided to kill that process and restarted the webserver. But still, accesssing the wiki was not possible: the page was "Loading...", but nothing got delivered back to the browser. Deleted cookies, tried another browser, still the wiki appeared to be unusable.
Looking at the MySQL database serving the wiki, I could see what was going on:
------ 307 Query REPLACE /* MessageBlobStore::updateModule 10.0.0.123 */ INTO `mw_msg_resource` (mr_resource,mr_lang,mr_blob,mr_timestamp) VALUES ('user.tokens','en','{}','20120926011737')
307 Query SELECT /* MessageBlobStore::updateModule 10.0.0.123 */ mr_blob FROM `mw_msg_resource` WHERE mr_resource = 'user.options' AND mr_lang = 'en' LIMIT 1 ------
These two queries where sent over and over again to the database by the php5-cgi process, ~10 times per second or so. Restarting the database did not help, as soon as I accessed the wiki again, these queries appeared again.
I've had this issues before, with 1.19.0, IIRC. I did not dig deeper though, somehow the next morning the wiki was "just" working again. But the issue was the same: changed something in the config and then these strange queries appeared.
I've put the LocalSettings.php and the database query log here:
https://trent.utfs.org/bits/mediawiki/
Some more notes:
* I've switched from lighttpd/php5-cgi to apache2/mod_php5, still the same: when the wiki is accessed, those strange queries appear and the wiki page "loads" until php's maximum execution time is exceeded..
* I'm using the APC cache. Disabling that (and wgMainCacheType) did no help either.
* I'm not using any Mediawiki extensions.
Anyone got an idea what might be going on here?
Thanks, Christian. -- BOFH excuse #148:
Insert coin for new game
_______________________________________________ MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
On Wed, 26 Sep 2012 at 00:55, Christian Kujau wrote:
- I've switched from lighttpd/php5-cgi to apache2/mod_php5, still the same: when the wiki is accessed, those strange queries appear and the wiki page "loads" until php's maximum execution time is exceeded..
To be clear, this happens in the log then:
[Thu Sep 27 00:11:35 2012] [error] [client 10.0.0.123] PHP Fatal error: Maximum execution time of 30 seconds exceeded in ../mediawiki/includes/db/Database.php on line 1941
or:
[Thu Sep 27 00:14:19 2012] [error] [client 67.164.57.135] PHP Fatal error: Maximum execution time of 30 seconds exceeded in ../mediawiki/includes/db/DatabaseMysql.php on line 31
Raising PHP's max_execution_time to e.g. 300s helps...sometimes. But I still have to wait >30s until this one wiki site is delivered, while mysqld is trying to serve these strange queries.
Anyone?
Thanks, Christian.
On Wed, 26 Sep 2012 at 00:55, Christian Kujau wrote:
307 Query REPLACE /* MessageBlobStore::updateModule 10.0.0.123 */ INTO `msg_resource` (mr_resource,mr_lang,mr_blob,mr_timestamp) VALUES ('user.tokens','en','{}','20120926011737')
307 Query SELECT /* MessageBlobStore::updateModule 10.0.0.123 */ mr_blob FROM `msg_resource` WHERE mr_resource = 'user.options' AND mr_lang = 'en' LIMIT 1
Looking at the database, the table msg_resource had 1,872,796 rows, using 126MB (out of 167 MB for the whole database, including msg_resource). This is a really small wiki, so I'm not sure why this table had so many records.
According to the documentation[0], this table is used to store "JSON message blobs for the resource loader", whatever that means. In my case, truncating msg_resource did the trick. The wiki is now up & running, msg_resource contains 36 rows.
For the curious mind, this is how it looked before:
mysql> select count(*) as a, mr_resource from msg_resource group by mr_resource order by a; [....] | 7040 | mediawiki.action.watch.ajax | | 7040 | mediawiki.api | | 7040 | mediawiki.api.watch | | 14400 | jquery.client | | 14400 | jquery.cookie | | 14400 | jquery.messageBox | | 14400 | jquery.mwExtension | | 14400 | mediawiki.legacy.ajax | | 14400 | mediawiki.legacy.wikibits | | 14400 | mediawiki.page.startup | | 14400 | mediawiki.util | | 23100 | jquery.makeCollapsible | | 23100 | jquery.mw-jump | | 23100 | jquery.placeholder | | 23100 | mediawiki.user | | 23100 | mediawiki.page.ready | | 23100 | jquery.checkboxShiftClick | | 884736 | user.options | | 884736 | user.tokens | +--------+--------------------------------+ 35 rows in set (2.42 sec)
I'll have to keep an eye on that table. While truncating helped in this case, this still smells like some kind of bug...
Thanks for listening, Christian.
[0] https://www.mediawiki.org/wiki/Manual:Msg_resource_table
mediawiki-l@lists.wikimedia.org