On Tue, Jun 24, 2008 at 8:57 PM, Travis (wikiHow) <travis(a)wikihow.com> wrote: Tim Starling would be the most knowledgeable, I guess. The major change in the 1.12 parser is the addition of the new preprocessor, and that specifically revamps replaceVariables() et al. However, on Wikimedia it showed a considerable *decrease* in the time taken to render pages. Perhaps there's a configuration difference, or it might be something else entirely that's causing it. You could try switching to Parser_OldPP, which should work correctly in 1.12, to see if that makes a difference. However, Parser_OldPP appears to no longer be maintained to any significant extent, so it probably isn't advisable to use it indefinitely, at least not in 1.13. The ParserDiffTest extension can be used to systematically compare the performance (and output) of the old and new parsers.

On Thu, Jun 26, 2008 at 12:02 PM, Travis (wikiHow) <travis(a)wikihow.com> wrote: I can't see why it would be. Are you running the two on the same machine? If so, is one of them being run against a production database and one against a testing database? If one is being used more than the other, that one is going to have more cache hits and work faster, so your production 1.9 site is of course going to run faster than your testing 1.12 site, even on the same server. Well, the current figure on Wikipedia seems to be under 10 queries per view: http://noc.wikimedia.org/cgi-bin/ng/report.py?db=enwiki So maybe you should be looking into what extra queries are being run. There are some features that can cause nasty query spam if things aren't configured right (like interwikis). Are you using MyISAM? If so, why? MediaWiki isn't really designed for it at all; it uses InnoDB-specific features like primary key clustering heavily. Why don't you use maintenance/update.php from a shell? It's almost certain to be more reliable. Not sure, that difference might be noise. Is it consistent? Make sure you repeat them both until you get a consistent figure, preferably on a server doing nothing else; otherwise you can get issues with caches getting messed up. Also, again, make sure that the databases are *both* cold, not production databases.

Do you think it might be an improvement if both Title::getParentCategories and Title::getParentCategoryTree could store their results in a local variable? They currently fetch the resuts DB each time regardless of whether they've been called already or not. I'm not sure how much this is called in a normal installation, but we call it a few times to figure out the category structure of the article. Storing the results seems to cut down our # of db calls by about 12 per page view.

On Thu, Jun 26, 2008 at 3:45 PM, Travis (wikiHow) <travis(a)wikihow.com> wrote: Yes, definitely! It will use the OS cache for the file blocks and the MyISAM key buffer for the key blocks. It will be working with hot data, so it will certainly be much faster. A typical view on your 1.9 instance might require no actual disk reads at all, while your 1.12 install could require many. A key buffer hit wouldn't even require a system call. This could easily account for a fourfold difference or more -- in principle, even a thousand-fold difference, although you aren't seeing that here (production cache might not be totally hot, dev cache might not be totally cold). You could try hitting the 1.12 install with artificial traffic to populate the caches, but it's still not necessarily going to behave quite the same. For performance benchmarking you really want exactly the same load patterns on everything you're testing. How far you want to go in achieving this is up to you. At a minimum I'd copy the 1.9 DB twice to two separate dev databases, and upgrade one of them. You could then do benchmarking with ab or a similar tool. Two significant things that come to mind in the core software config: one, if you're running a large number of interwikis, you need special configuration for that to avoid a ton of DB traffic. Two, you should set $wgJobRunRate = 0 and have the job queue run using maintenance/runJobs.php in cron, not on each page view. It's almost certainly a bogus statistic. It could just reflect a long time waiting for a lock or something. I'd imagine that would be a good idea, yes. Make sure you reconfigure MySQL, though! Standard settings are totally unacceptable for heavy InnoDB use. You'll want to set key_buffer_size smaller and increase innodb_buffer_pool_size to most of your database server's available memory (if no serious MyISAM apps are running on the DB server -- and anyway temporary tables are always MyISAM, so don't set key_buffer_size too small!). This is a good article to read: http://www.mysqlperformanceblog.com/2007/11/01/innodb-performance-optimizat… It's covered in the manual: http://www.mediawiki.org/wiki/Manual:Upgrading_to_1.12

On Thu, Jun 26, 2008 at 4:51 PM, Travis (wikiHow) <travis(a)wikihow.com> wrote: Well, I should clarify that spending a long time waiting for a lock doesn't mean there's no problem, it just means that the problem isn't with the particular query. Switching to InnoDB might help with that a lot, if it is a locking issue, since reads don't normally do any locking. I can't claim to have a very good idea of what that method does, in practice. It looks quite complicated. Tim Starling has pointed out on this list that it makes no difference. Contrary to the manual, you don't seem to need to manually free results, at least not for memory-usage purposes. This code uses steadily more memory until it OOMs: $enormous = array(); while( true ) { $enormous []= $dbr->select( 'user', '*', array( 'user_id' => 1 ) ); } But this uses practically no memory: while( true ) { $result = $dbr->select( 'user', '*', array( 'user_id' => 1 ) ); } You might want to cache it in your skin, then. Although if it's just $wgTitle, then per-object Title-level caching should be equally effective and not an overly large memory hog.

Travis (wikiHow) wrote: $wgProxyKey has nothing to do with memcached; the only thing it's used for is the remote HTTP proxy scanning feature which is disabled by default and probably doesn't work right. -- brion