Hi all,
Ever since my recent upgrades to MediaWiki 1.27, which included upgrading the operating system and changing from Apache + mod_php to Apache + PHP-FPM, a few times I've seen PHP-FPM processes hanging on a certain kind of old page revision (often these revisions are years old). Specifically, it appears to be old page revisions where there was a huge change to the page size, e.g. a 64k page having 61k added to it. Here's an example excerpt from the FPM status page with the pool info and one of the hung processes:
pool: www process manager: dynamic start time: 08/Dec/2016:07:59:44 -0800 start since: 1041140 accepted conn: 10654252 listen queue: 0 max listen queue: 0 listen queue len: 0 idle processes: 7 active processes: 12 total processes: 19 max active processes: 40 max children reached: 8 slow requests: 0
pid: 7087 state: Running start time: 20/Dec/2016:08:36:30 -0800 start since: 2134 requests: 38 request duration: 2088600826 request method: GET request URI: /index.php?title=Chef&oldid=363770 content length: 0 user: - script: /var/www/sites/gw2w-en/index.php last request cpu: 0.00 last request memory: 0
Currently php.ini has max_execution_time set to 60s. I realize I could also try setting the pool's request_terminate_timeout value to ensure processes don't stay hung like this, but that's basically kill -9 and I'd rather try to resolve the underlying issue and improve the performance so these old revisions can actually load in a reasonable amount of time.
This is on up-to-date Ubuntu 16.04.3 servers, with MediaWiki 1.27.1, Apache 2.4.18, and PHP 7.0.8. Also for what it's worth, the referrers for these page requests have pretty much always been http://www.bing.com/bingbot.htm, but since this is easily reproducible, I'd rather fix the problem so actual users aren't forced to wait several minutes for old revision page loads.
Any suggestions?
Thanks, Justin
Hi,
On 12/20/2016 09:28 AM, Justin Lloyd wrote:
Hi all,
Ever since my recent upgrades to MediaWiki 1.27, which included upgrading the operating system and changing from Apache + mod_php to Apache + PHP-FPM, a few times I've seen PHP-FPM processes hanging on a certain kind of old page revision (often these revisions are years old). Specifically, it appears to be old page revisions where there was a huge change to the page size, e.g. a 64k page having 61k added to it. Here's an example excerpt from the FPM status page with the pool info and one of the hung processes:
Do you have the wikidiff2 PHP extension[1] installed? If you're having performance problems with diffs, switching to the C-based implementation should help. Diffs of that size are going to be large regardless, so I'm not too surprised it's having issues...
I'm not familiar with php-fpm timeouts to give advice on that, sorry.
[1] https://www.mediawiki.org/wiki/Extension:Wikidiff2
-- Legoktm
Hi,
No, I wasn't even aware of it. Searching my systems for wikidiff only turns up includes/diff/WikiDiff3.php in the MW code. I can try testing wikidiff2 by installing the php-wikidiff2 package and seeing if that makes a difference.
Thanks, Justin
-----Original Message----- From: MediaWiki-l [mailto:mediawiki-l-bounces@lists.wikimedia.org] On Behalf Of Legoktm Sent: Tuesday, December 20, 2016 11:01 AM To: mediawiki-l@lists.wikimedia.org Subject: Re: [MediaWiki-l] PHP-FPM processes hanging on some large diff page revisions
Hi,
On 12/20/2016 09:28 AM, Justin Lloyd wrote:
Hi all,
Ever since my recent upgrades to MediaWiki 1.27, which included upgrading the operating system and changing from Apache + mod_php to Apache + PHP-FPM, a few times I've seen PHP-FPM processes hanging on a certain kind of old page revision (often these revisions are years old). Specifically, it appears to be old page revisions where there was a huge change to the page size, e.g. a 64k page having 61k added to it. Here's an example excerpt from the FPM status page with the pool info and one of the hung processes:
Do you have the wikidiff2 PHP extension[1] installed? If you're having performance problems with diffs, switching to the C-based implementation should help. Diffs of that size are going to be large regardless, so I'm not too surprised it's having issues...
I'm not familiar with php-fpm timeouts to give advice on that, sorry.
[1] https://www.mediawiki.org/wiki/Extension:Wikidiff2
-- Legoktm
_______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
This actually seemed to cause problems. After installing and enabling the wikidiff2 php module in my test environment, I would suddenly get server errors for pages that normally worked fine, so I don't think this is going to help anything.
For now I've set the PHP-FPM pool's request_slowlog_timeout to 60s and request_terminate_timeout to 2m. I haven't yet seen any log entries from processes being killed due to being slow, but I'll keep my eyes on it.
Justin
-----Original Message----- From: MediaWiki-l [mailto:mediawiki-l-bounces@lists.wikimedia.org] On Behalf Of Justin Lloyd Sent: Tuesday, December 20, 2016 11:18 AM To: MediaWiki announcements and site admin list mediawiki-l@lists.wikimedia.org Subject: Re: [MediaWiki-l] PHP-FPM processes hanging on some large diff page revisions
Hi,
No, I wasn't even aware of it. Searching my systems for wikidiff only turns up includes/diff/WikiDiff3.php in the MW code. I can try testing wikidiff2 by installing the php-wikidiff2 package and seeing if that makes a difference.
Thanks, Justin
-----Original Message----- From: MediaWiki-l [mailto:mediawiki-l-bounces@lists.wikimedia.org] On Behalf Of Legoktm Sent: Tuesday, December 20, 2016 11:01 AM To: mediawiki-l@lists.wikimedia.org Subject: Re: [MediaWiki-l] PHP-FPM processes hanging on some large diff page revisions
Hi,
On 12/20/2016 09:28 AM, Justin Lloyd wrote:
Hi all,
Ever since my recent upgrades to MediaWiki 1.27, which included upgrading the operating system and changing from Apache + mod_php to Apache + PHP-FPM, a few times I've seen PHP-FPM processes hanging on a certain kind of old page revision (often these revisions are years old). Specifically, it appears to be old page revisions where there was a huge change to the page size, e.g. a 64k page having 61k added to it. Here's an example excerpt from the FPM status page with the pool info and one of the hung processes:
Do you have the wikidiff2 PHP extension[1] installed? If you're having performance problems with diffs, switching to the C-based implementation should help. Diffs of that size are going to be large regardless, so I'm not too surprised it's having issues...
I'm not familiar with php-fpm timeouts to give advice on that, sorry.
[1] https://www.mediawiki.org/wiki/Extension:Wikidiff2
-- Legoktm
_______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l _______________________________________________ MediaWiki-l mailing list To unsubscribe, go to: https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
mediawiki-l@lists.wikimedia.org