We've twice tried to upgrade our MediaWiki site (www.uesp.net) running MediaWiki 1.14 to 1.19 but had the same problem each time. The basic upgrade process and resulting issue goes like:
- Set wiki to read only mode. - Backup database and copy into a new database. - Upgrade the new database to 1.19 (no issues though it takes an hour). - Test 1.19 wiki (no issues seen including aggressive load testing with ab). - Switch 1.19 to "live" site. - Right away the database server begins to be overloaded with hundreds of "DELETE FROM l10n_cache WHERE lc_lang=en;" queries. Some page views get through but eventually the Apache servers fill up with connections waiting for the database server which also fills up with connections that never terminate. - Switch back the live site to 1.14. The database server has to be forced closed and restarted to restore service (the delete queries never finish).
A few possible causes: - One of our extensions or custom code changes is causing an issue. Unfortunately, some of the more complex/custom extensions cannot be disabled without breaking the site. - Increased database load: Our db servers are very underused and a conservative estimate is they would work fine with a 20x load increase. A small load increase from the apparent changes between 1.14 and 1.19 shouldn't cause this level of database issue. - Our database was originally created with a very old MediaWiki (1.3 I think). While the upgrade from 1.14 to 1.19 works perhaps there are lingering issues that need the wiki database to be recreated from scratch somehow. - An issue with the MediaWiki localization db cache when used with a high traffic site.
My next guess on what to try is to set "$wgCacheDirectory" to a local directory which should completely avoid using the database for any localization messages if I understand things correctly. Unfortunately, I have no way of testing if this works without running through another complete upgrade attempt.
Any suggestions or ideas on what to try would be appreciated....
------------------------ Dave Humphrey - dave@uesp.net Server Admin - the Unofficial Elder Scrolls Pages - www.uesp.net
On Sat, Sep 1, 2012 at 10:37 AM, Dave Humphrey dave@uesp.net wrote:
My next guess on what to try is to set "$wgCacheDirectory" to a local directory which should completely avoid using the database for any localization messages if I understand things correctly.
Yes, that's right.
On 02/09/12 00:37, Dave Humphrey wrote:
- An issue with the MediaWiki localization db cache when used with a high
traffic site.
The DB store for the localisation cache is not really intended for high-traffic sites, it's intended to make small wikis work out of the box without extra configuration. For a high traffic site you should set $wgCacheDirectory to a writable directory, or better still, set $wgLocalisationCacheConf['manualRecache'] to true and run maintenance/rebuildLocalisationCache.php each time the code is updated.
To solve the general problem of cache stampedes, you may want to install the PoolCounter extension:
https://www.mediawiki.org/wiki/Extension:PoolCounter
-- Tim Starling
Changing to use the cache directory did indeed work fine, and thanks for the note about the PoolCounter which we'll look into.
On 2 September 2012 19:52, Tim Starling tstarling@wikimedia.org wrote:
On 02/09/12 00:37, Dave Humphrey wrote:
- An issue with the MediaWiki localization db cache when used with a high
traffic site.
The DB store for the localisation cache is not really intended for high-traffic sites, it's intended to make small wikis work out of the box without extra configuration. For a high traffic site you should set $wgCacheDirectory to a writable directory, or better still, set $wgLocalisationCacheConf['manualRecache'] to true and run maintenance/rebuildLocalisationCache.php each time the code is updated.
To solve the general problem of cache stampedes, you may want to install the PoolCounter extension:
https://www.mediawiki.org/wiki/Extension:PoolCounter
-- Tim Starling
MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
mediawiki-l@lists.wikimedia.org