Hello, I understand the need for cite, thats why it is still there :) But... (...)

Unfortunately, {{cite}} is the only template I can profile/account for now, we don't have proper per-template profiling, but I wish to get one some day. Then we'd have more "war on ..." topics ;-D

Generally, templates are major part of our parsing, and thats over 50% of our current cluster CPU load.

Hello, I understand the need for cite, thats why it is still there :) But... - We format Cite references list every 100th request to backend, though it takes 8.15% backend response time (thanks parser cache, without it Cite formatting would take 815% cluster time - though developers should understand I'm not exactly right at this hyperbole ;-) - When parsing articles like one of most popular today, [[en:Rod_Blagojevich_corruption_charges]], it takes 20s to produce the page, 17s is spent on Cite block, executing {{cite}} mostly. That makes every editor wait for ages to get a page displayed, and due to cache stampede after invalidation it causes considerable stress on site (look at numbers mentioned above). - This 8% is in real-time, which includes waiting for search, databases, and simply CPU contention, which we end up having today. CPU-time wise it is way higher, so can actually have 20% CPU time impact on our application farm. Thats at least 100k$ worth of hardware (and rising), even if new/modern one, just for citation formatting. So, a checklist what can be done ( simple to complex ) [ ] - Simplification of {{cite}} [ ] - Separate cache for Cite, to avoid reparsing on minor edits, that don't involve citations. I have no idea how much this would win, but there is theoretical chance of stripping 1% or so. ;) [ ] - Offload some templates like {{cite}} to actual PHP extensions (can of worms, but, oh well, can be standardized process too) [ ] - Implement proper scripting engine like Lua for metatemplates (http://pecl.php.net/package/lua - another can of worms, though yet again, can be managed via trusted set of people, on top20 wikis or so). [ ] - Frustrated operations guy adding something like ( return ""; ) in some random extension, and syncing the live hack. Obviously there would be some "HAHA YOU THOUGHT I COULDN'T DO THIS" comments in there. I for one can directly participate in at least two of these options. ;-) Unfortunately, {{cite}} is the only template I can profile/account for now, we don't have proper per-template profiling, but I wish to get one some day. Then we'd have more "war on ..." topics ;-D Generally, templates are major part of our parsing, and thats over 50% of our current cluster CPU load. As we've actually managed to hit 100% last week, something what hasn't happened for a while, some of work has to be done here. Of course, new hardware will help for a while, but I for one have huge personal satisfaction saving donation money. ;-) CHEERS! -- Domas Mituzas -- http://dammit.lt/ -- [[user:midom]] _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

A long while ago I remember looking at the parser and realizing that the recursive template expansion and argument handling led the parser to run all branches of #if and #switch statements before deciding which one to include. In other words, given {{#if: something | statements_A | statements_B }}, the parser was fully expanding both statements_A and statements_B before checking #if to decide which one to keep. Obviously that is inefficient and in the case of very complicated conditional templates potentially very expensive.

On Sat, Jan 31, 2009 at 1:28 PM, Alex <mrzmanwiki(a)gmail.com> wrote: What about throwing them in MediaWiki: space, similar to editnotices? At least then they could be cached to hell and back in the message cache. -Chad

Chad wrote: I considered that as well, but I'm not sure how much that will actually help. Looking at http://en.wikipedia.org/wiki/Joe%20the%20Plumber?action=purge&forceprof… it took 21.796 seconds to load, most of which seems be from Parser::recursiveTagParse, about 90% of that that is from Cite::referencesFormat-parse. Even if the templates themselves are heavily cached, it still has to run all the conditionals and formatting through the parser. Heavy caching might help if there's lots of refs with the same content on multiple pages, but I don't think that's very common. -- Alex (wikipedia:en:User:Mr.Z-man) _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

On Sat, Jan 31, 2009 at 8:03 AM, Domas Mituzas <midom.lists(a)gmail.com> wrote: This seems like it's the only solution from your list that would be generally applicable to similar future scenarios. I don't think the users would have to be particularly trusted -- just make sure that the runtime of the programs is limited, and that it's properly sandboxed (is the Lua PECL extension sandboxed?).

On Sat, Jan 31, 2009 at 8:19 PM, Platonides <Platonides(a)gmail.com> wrote: Yes, that would be the major disadvantage I can see. In practice, nobody can reuse large chunks of Wikipedia content on shared hosting anyway, since it's way too big, but it would be a serious obstacle for people who want to reuse only parts of Wikipedia.

A long while ago I remember looking at the parser and realizing that the recursive template expansion and argument handling led the parser to run all branches of #if and #switch statements before deciding which one to include. In other words, given {{#if: something | statements_A | statements_B }}, the parser was fully expanding both statements_A and statements_B before checking #if to decide which one to keep. Obviously that is inefficient and in the case of very complicated conditional templates potentially very expensive. The parser has changed so much since I last worked with it that I am having difficulty figuring out if this is still true. Hopefully, someone already went through and improved the branch handling logic, but if not, I would suggest that this would also be a good generalized target for improving template operation.

- When parsing articles like one of most popular today, [[en:Rod_Blagojevich_corruption_charges]], it takes 20s to produce the page, 17s is spent on Cite block, executing {{cite}} mostly. That makes every editor wait for ages to get a page displayed, and due to cache stampede after invalidation it causes considerable stress on site (look at numbers mentioned above).

Strictly speaking it should be the editors' job to properly close tables and divs, etc., but because Tidy is so good at it they don't have to, which makes our wikicode less portable. -Robert Rohde

[ ] - Separate cache for Cite, to avoid reparsing on minor edits, that don't involve citations. I have no idea how much this would win,

in few minutes Cite cache dropped Cite out of our profiling top50 page, and even though it accounted for 10% of cluster load today, it is under 3% already. I claim 90% of honor for this improvement, the rest goes to Andrew, who implemented it all ;-D Do note, the war is not over yet, we will have to think how to resolve metatemplates properly :) BR, -- Domas Mituzas -- http://dammit.lt/ -- [[user:midom]] _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l