Please copy this to your local village pump or other relevant on-wiki forum.
Werdna's #ifexist limit feature is now live. In response to complaints of template breakage, I have increased the limit on Wikimedia wikis temporarily, from 100 to 2000. Barring a coup, it will stay at 2000 for about a week, and then we'll lower it to 100.
Please use this one-week period to check pages and templates that use #ifexist heavily. Look in the HTML source of the preview or page view. There will be a "limit report" that looks like this:
<!-- Pre-expand include size: 617515/2048000 bytes Post-expand include size: 360530/2048000 bytes Template argument size: 51168/2048000 bytes #ifexist count: 1887/2000 -->
This is the limit report from http://commons.wikimedia.org/wiki/Template:Potd/2007-12 , one of the pages that will break.
At the end of the week, any pages which have a #ifexist count of over 100 will cease to be rendered correctly (after the next edit or cache clear). All #ifexist calls after the hundredth will be treated as if the target does not exist.
In some cases it may be possible to rewrite your templates so that they still do the same thing, but with less #ifexist calls. In other cases, you will need to remove template features. Removing features is always sad, as a sofware developer I know that, but sometimes it is necessary for the good of the project. This is one of those times.
-- Tim Starling
Tim Starling wrote:
Please copy this to your local village pump or other relevant on-wiki forum.
Werdna's #ifexist limit feature is now live. In response to complaints of template breakage, I have increased the limit on Wikimedia wikis temporarily, from 100 to 2000. Barring a coup, it will stay at 2000 for about a week, and then we'll lower it to 100.
Please use this one-week period to check pages and templates that use #ifexist heavily. Look in the HTML source of the preview or page view. There will be a "limit report" that looks like this:
<!-- Pre-expand include size: 617515/2048000 bytes Post-expand include size: 360530/2048000 bytes Template argument size: 51168/2048000 bytes #ifexist count: 1887/2000 -->
This is the limit report from http://commons.wikimedia.org/wiki/Template:Potd/2007-12 , one of the pages that will break.
At the end of the week, any pages which have a #ifexist count of over 100 will cease to be rendered correctly (after the next edit or cache clear). All #ifexist calls after the hundredth will be treated as if the target does not exist.
In some cases it may be possible to rewrite your templates so that they still do the same thing, but with less #ifexist calls. In other cases, you will need to remove template features. Removing features is always sad, as a sofware developer I know that, but sometimes it is necessary for the good of the project. This is one of those times.
-- Tim Starling
I suggest to cache these calls results (as done with "#time"), and to count multiple calls to the same page as one call, in case someone wants to check the same page multiple times. (Title objects are cached anyway, and so are the article IDs, so they don't have to cause multiple queries anyway; they are harder to check, though.) "Then" and "else" should not be cached, just the title and its result.
Rotem Liss wrote:
I suggest to cache these calls results (as done with "#time"), and to count multiple calls to the same page as one call, in case someone wants to check the same page multiple times. (Title objects are cached anyway, and so are the article IDs, so they don't have to cause multiple queries anyway; they are harder to check, though.) "Then" and "else" should not be cached, just the title and its result.
+1 In my little journey in the #ifexist abuse i have found that people use this kind of constructs quite frequently:
Lorem ipsum (A ? <strong>: <small>) dolor (A ? </strong>:</small>) sic amet consectetuer adipiscing elit. (A ? "Proin quam. Praesent vulputate libero eu arcu." : "") Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. (A ? "Pellentesque vitae diam." : ) Proin elementum, purus eu rhoncus rhoncus, lacus nisi hendrerit metus, et congue nunc ante vel odio.
So it's fixed doing: if (A) {
Lorem ipsum <strong>dolor</strong> sic amet consectetuer adipiscing elit. Proin quam. Praesent vulputate libero eu arcu. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Pellentesque vitae diam. Proin elementum, purus eu rhoncus rhoncus, lacus nisi hendrerit metus, et congue nunc ante vel odio.
} else {
Lorem ipsum <small>dolor</small> sic amet consectetuer adipiscing elit. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Pellentesque vitae diam. Proin elementum, purus eu rhoncus rhoncus, lacus nisi hendrerit metus, et congue nunc ante vel odio.
}
Which decreases the count incredibly (put that on a template called with different arguments a lot of times from another template used on many pages). So caching the answer for each checked title on the same call will make a difference (and is the sensible thing to do: nobody expects it to return different on the same page, though the double race condition exists). Maybe for clarity split them into '#ifexist count' and 'Active #ifexist count'?
List of pages with over 100 #ifexist calls (warning, large file):
http://noc.wikimedia.org/~tstarling/ifexist.log
-- Tim Starling
On 12/1/07, Tim Starling tstarling@wikimedia.org wrote:
List of pages with over 100 #ifexist calls (warning, large file):
For technical reasons, I didn't finish waiting for this page to load, but I did notice several repetitions of the urls for a relatively small number of templates. I suspect that a list filtered to exclude pages outside namespace 10 (and duplicate urls for that matter) would be more browser-friendly and closer to the source of the problem.
—C.W.
On 12/4/07, Charlotte Webb charlottethewebb@gmail.com wrote:
On 12/1/07, Tim Starling tstarling@wikimedia.org wrote:
List of pages with over 100 #ifexist calls (warning, large file):
For technical reasons, I didn't finish waiting for this page to load, but I did notice several repetitions of the urls for a relatively small number of templates. I suspect that a list filtered to exclude pages outside namespace 10 (and duplicate urls for that matter) would be more browser-friendly and closer to the source of the problem.
It seems to include random globs of binary data, like a big chunk of nulls from 0x5004 to 0x54D, another at 0x1100E to 0x117C5, etc. There's over a megabyte of them total in the 13 MB file when I wget it. (Maybe my downloads were corrupted, though.) But you have weirdness like:
enwiki: http://en.wikipedia.org/wiki/Template:Archive_box 137 enwiki: http://en.wikipedia.org/wiki/Template:Archive_box 1675 enwiki: http://en.wikipedia.org/wiki/Template:Archive_box 1775 enwiki: http://en.wikipedia.org/wiki/Template:Archive_box 259 enwiki: http://en.wikipedia.org/wiki/Template:Archive_box 359 enwiki: http://en.wikipedia.org/wiki/Template:Archive_box 475 enwiki: http://en.wikipedia.org/wiki/Template:Archive_box 575 enwiki: http://en.wikipedia.org/wiki/Template:Archive_box 859 enwiki: http://en.wikipedia.org/wiki/Template:Archive_box 959
which gives nine distinct {{#ifexist:}} counts for the same template -- and 859 total instances of it.
The same file with all nulls removed, restricted only to Template:, is under 100 KB. I've uploaded it here:
http://twcenter.net/~simetrical/ifexist-truncated.log
Of course I'm not sure what to make of some of the duplicates, but either way, things like Template:Archive_box need to be fixed, evidently. And probably a lot of the non-template pages too, that include multiple {{#ifexist:}} templates or that trigger paths not executed on direct view or whatever.
I have made the file with duplicates removed and only the titles shown (not the numbers). It can be found at http://hyperfileshare.com/d/bce7ef1d.
The following wikis have the following number of pages that will get problems:
afwiki: 19 anwiki: 1 arwiki: 20 be_x_oldwiki: 1 bgwiki: 6 bnwiki: 6 brwiki: 6 bswiki: 3 bswiktionary: 1 cawiki: 172 commonswiki: 36 cswiki: 113 cswikisource: 1 dawikibooks: 2 dewiki: 239 dewikibooks: 10 dewikiquote: 1 dewikisource: 228 dewikiversity: 15 dewiktionary: 4 emlwiki: 1 enwiki: 12917 enwikibooks: 67 enwikinews: 2 enwikiquote: 1 enwiktionary: 103 eowiki: 141 eswiki: 158 eswikibooks: 8 euwiki: 1 fawiki: 1 fiwiki: 1 frwiki: 1540 glwiki: 2 hewiki: 13 hewikisource: 15 hiwiki: 3 hrwiki: 91 hsbwiki: 542 huwiki: 196 iawiki: 1 idwiki: 6 incubatorwiki: 1 itwiki: 356 itwikibooks: 2 jawiki: 4 jawikibooks: 1 kawiki: 8 kkwiki: 9 kshwiki: 1 lbewiki: 7 lvwiki: 203 mediawikiwiki: 5 metawiki: 557 mkwiki: 13 mlwiki: 1 mswiki: 19 newwiki: 3 nlwiki: 35 nnwiki: 1 nowiki: 31 oswiki: 7 plwiki: 105 plwikinews: 1 plwiktionary: 1 pswiki: 1 ptwiki: 83 ptwiktionary: 21 quwiki: 7 rowiki: 64 rowiktionary: 830 ruwiki: 4064 sdwiktionary: 1 simplewiki: 54 skwiki: 20 slwiki: 26 sqwiki: 1 srwiki: 3 svwiki: 1 tawiki: 2 tewiki: 17 thwiki: 56 trwiki: 286 ukwiki: 64 vowiki: 2 zh_min_nanwikibooks: 1 zh_yuewiki: 1 zhwiki: 1108
It seems to include random globs of binary data, like a big chunk of nulls from 0x5004 to 0x54D, another at 0x1100E to 0x117C5, etc. There's over a megabyte of them total in the 13 MB file when I wget it. (Maybe my downloads were corrupted, though.) But you have weirdness like:
thats how NFS appends work :)
On 12/6/07, Domas Mituzas midom.lists@gmail.com wrote:
It seems to include random globs of binary data, like a big chunk of nulls from 0x5004 to 0x54D, another at 0x1100E to 0x117C5, etc. There's over a megabyte of them total in the 13 MB file when I wget it. (Maybe my downloads were corrupted, though.) But you have weirdness like:
thats how NFS appends work :)
Maybe, but it makes it rather inconvenient to open the file in a text editor without tr -d "\0".
The #ifexist limit has now been reduced to 500 unique calls. A list of broken pages appears at:
[[Category:Pages with too many ifexist calls]]
or its equivalent localised via pfunc_max_ifexist_category.
We're still running on Parser_OldPP, more bugs found.
-- Tim Starling
wikitech-l@lists.wikimedia.org