I added the following regarding 'pcre.backtrack_limit' and PHP 5.2.x to the MediaWiki.org wiki at http://www.mediawiki.org/wiki/Manual:$wgSpamRegex :
===pcre.backtrack_limit===
PHP 5.2.x introduced pcre.backtrack_limit with default 100000 (less than 100K). I think that is too low and trips up the regex. See stronk7 at moodle dot org's 13-Sep-2007 comment (Find '13-Sep-2007') at http://us.php.net/manual/en/ref.pcre.php. Try adding the following line to LocalSettings.php:
ini_set( 'pcre.backtrack_limit', '2M' );
I don't know what value is appropriate. 2M works for me. Someone who knows more please adjust that.
Is 2M reasonable there? What about 8M?
What would be a safe setting? What other server configs dictate what is appropriate there?
And I set 'memory_limit' to 20M in my LocalSettings.php. Is that necessary? Is this appropriate, too?
ini_set( 'memory_limit', '20M' );
What sort of trouble am I getting into?
Thanks,
Roger --Wikigogy.org MediaWiki 1.16.0 PHP 5.2.14 (litespeed) MySQL 5.0.91-community-log
Current versions of MediaWiki (1.16+) will actually check the memory limit on every pageview and raise it if it's below 50MB, by default. (The installer used to add an ini_set line in LocalSettings.php; see http://www.mediawiki.org/wiki/Manual:$wgMemoryLimit ). There shouldn't be any need to futz with it.
What exactly leads you to suspect pcre.backtrack_limit is causing problems? Do you have a huge regex that's being silently ignored, or are you getting an error?
On Tue, Nov 9, 2010 at 10:38 AM, Benjamin Lees emufarmers@gmail.com wrote:
Current versions of MediaWiki (1.16+) will actually check the memory limit on every pageview and raise it if it's below 50MB, by default. (The installer used to add an ini_set line in LocalSettings.php; see http://www.mediawiki.org/wiki/Manual:$wgMemoryLimit ). There shouldn't be any need to futz with it.
What exactly leads you to suspect pcre.backtrack_limit is causing problems? Do you have a huge regex that's being silently ignored, or are you getting an error?
I don't know where to look for errors about this. Is there is a log, maybe a PHP log, that would record when a $wgSpamRegex runs out of memory?
Here's the evidence I have that it might be running out of memory:
With 'pcre.backtrack_limit' at default (not set in LocalSettings.php, and 'memory_limit' not set either) a clumsy regex like this one (meant to match external links not separated by at least two words), $wgSpamRegex = "/http://\S*\s*\S*\s*\S*\s*\S*http://\S*/i";
..fails to match any of the following wiki edit field text:
==Banana== [http://link.link.banababot.com/linkety-link:Linkety-plonk-1 Link-1] b a n a n a [http://link.link.banababot.com/linkety-link:Linkety-plonk-2 Link-2] [http://link.link.banababot.com/linkety-link:Linkety-plonk-3 Link-3]
..However, it match links 2 and 3 when they are on top, as in this edit text: ==Banana== [http://link.link.banababot.com/linkety-link:Linkety-plonk-2 Link-2] [http://link.link.banababot.com/linkety-link:Linkety-plonk-3 Link-3] [http://link.link.banababot.com/linkety-link:Linkety-plonk-1 Link-1] b a n a n a
Links 2 and 3 should match whether or not link 1 is above them. If I set 'pcre.backtrack_limit' to 8M, or even 2M, then the regex successfully matches links 1 and 2 in both texts above, as intended. With 'pcre.backtrack_limit' at its default setting, not set in LocalSettings.php, the regex fails to match the first text above. I think it runs out of memory because it is a _clumsy_ regex. Where might I look for a log of it failing or running out of memory?
The following regex is _better formed_, I think, and successfully matches both above texts with or without a 'pcre.backtrack_limit' setting in LocalSettings.php:
$wgSpamRegex = "http://(\S+\s+){1,3}\S*http://\S+|"/i; # Matches 2 external links with less than " x.. x.. " between them
I did the above tests by editing LocalSettings.php and then trying test edits in my wiki.
What is the relationship and precedence between PHP's 'pcre.backtrack_limit' and 'memory_limit' settings?
Thanks for your help!
Roger --Wikigogy.org MediaWiki 1.16.0 PHP 5.2.14 (litespeed) MySQL 5.0.91-community-log
Oops, for my better formed regex lose the "|" at the end, like so:
# Matches 2 external links with less than " x.. x.. " between them: $wgSpamRegex = "http://(\S+\s+){1,3}\S*http://\S+"/i;
Unless you have it as part of a larger regex of or "|" parts, which I actually do, on several lines concatenated like so (with much #commented out):
$wgSpamRegex = "/". # "/" is the opening wrapper #"s-e-x|zoofilia|sexyongpin|grusskarte|geburtstagskarten|animalsex|". #"sex-with|dogsex|adultchat|adultlive|camsex|sexcam|livesex|sexchat|". #"chatsex|onlinesex|adultporn|adultvideo|adultweb.|hardcoresex|hardcoreporn|". #"teenporn|xxxporn|lesbiansex|livegirl|livenude|livesex|livevideo|camgirl|". #"spycam|voyeursex|casino-online|online-casino|kontaktlinsen|cheapest-phone|". #"laser-eye|eye-laser|fuelcellmarket|lasikclinic|cragrats|parishilton|". #"paris-hilton|paris-tape|2large|fuel-dispenser|fueling-dispenser|huojia|". #"jinxinghj|telematicsone|telematiksone|a-mortgage|diamondabrasives|". #"reuterbrook|sex-plugin|sex-zone|lazy-stars|eblja|liuhecai|". #"buy-viagra|-cialis|-levitra|boy-and-girl-kissing|". # Match spammy words #"dirare.com|". # Matches dirare.com a spammer's domain name "overflow\s*:\s*auto|". # Matches overflow:auto regardless of whitespace "height\s*:\s*[0-4]px|". # Matches height:0px (most CSS hidden spam) "<\s*a\s*href|". # Matches '<a href' links, forcing wiki syntax #"(http://(.|\n)*){14}|". # Matches x number of external links #"http://\S*\s*\S*\s*\S*\s*\S*\s*\S*\s*\S*http://\S*|". # Roger -- bad #"http://\S*\s*\S*\s*\S*\s*\S*http://\S*|". # Roger -- NB pcre.backtrack_limit "http://(\S+\s+){1,3}\S*http://\S+|". # Roger -- better! "display\s*:\s*none". # Matches display:none regardless of whitespace "/i"; # "/" ends the regular expression, "i" case-insensitive # "\s" matches whitespace # "*" is a repeater (zero or more times) # "\s*" means to look for 0 or more amount of whitespace # "\S*" means to look for 0 or more amount of non-whitespace
Largely copied from http://www.mediawiki.org/wiki/Manual:$wgSpamRegex#A_Large_Example
Roger
On Tue, Nov 9, 2010 at 8:41 PM, roger@rogerchrisman.com wrote:
Links 2 and 3 should match whether or not link 1 is above them. If I set 'pcre.backtrack_limit' to 8M, or even 2M, then the regex successfully matches links 1 and 2 in both texts above, as intended. With 'pcre.backtrack_limit' at its default setting, not set in LocalSettings.php, the regex fails to match the first text above. I think it runs out of memory because it is a _clumsy_ regex. Where might I look for a log of it failing or running out of memory?
It doesn't sound like it's running out of memory: backtrack_limit is doing its job and preventing the clumsy regex from eating up the system's resources. Out-of-memory errors will be reported in your PHP error log or your webserver error log; backtrack_limit hits would presumably be logged there as well, although the comment you linked to complains that they're not.
The moral here is probably that you shouldn't use wasteful regexes (but you can raise the limit if you must).
What is the relationship and precedence between PHP's 'pcre.backtrack_limit' and 'memory_limit' settings?
memory_limit is not your problem. (The line you added in LocalSettings.php isn't even having any effect, since Setup.php raises the limit after LocalSettings.php is loaded.)
mediawiki-l@lists.wikimedia.org