The larger time range includes the removal of the mysql based parsercache which is the cause of the primary decline and not related to 1.19. The time range of the originally mentioned graph just shows a bit of context before the enwiki deployment up until the time I posted it to irc last night.
A key change as hashar suggests is a good theory but I'm not sure if the hit rate is actually recovering when looking at -24 or -8 hours, more time will tell.
This may not be valid since the mysql pcache wasn't re-enabled (from an empty state) long before 1.19 but the rate of selects against it seems to be up day over day. Maybe a new key is consistently fetched before it would ever be set.
2012-02-29T11:24:00+00:00,10288.770238 2012-02-29T12:06:00+00:00,10992.754365 2012-02-29T12:48:00+00:00,11140.606746
2012-03-01T11:54:00+00:00,13912.613492 2012-03-01T12:36:00+00:00,13790.359524 2012-03-01T13:18:00+00:00,14176.010317
(from http://ganglia.wikimedia.org/latest/graph.php?c=MySQL%20pmtpa&h=db40.pmt... )
On Thursday, March 1, 2012, Antoine Musso wrote:
Le 01/03/12 09:50, Jeroen De Dauw a écrit :
Hey,
There's been a slight regression in our parser cache hit rate:
This one is probably more informative for people not aware of the usual hit rate http://goo.gl/YY80C
Looks to me that the miss rate went up over 500% - is that really just a "slight" regression? :)
We really want to use absolute time range: http://bit.ly/A7kcys
Anyway, they are absent misses. Probably a key changed somewhere in our parser that magically invalidated roughly 15% of the parser cache. It seems to slowly recover afterward.
By zooming and making the Y scale start at 50%, the event seems to have occurred on February 17th just before 6am UTC.
I have uploaded a screenshot on mediawiki.org :
https://www.mediawiki.org/**wiki/File:Parser_cache_hit_**20120217.pnghttps://www.mediawiki.org/wiki/File:Parser_cache_hit_20120217.png
From the wikitech admin logs we have:
07:42 binasher:upgraded mysql on db40 to 5.1.53-facebook-r3753, enabled innodb_use_purge_thread ------------------------------**------------------------------**---------- 05:39 tstarling synchronized wmf-config/InitialiseSettings.**php 05:38 tstarling synchronized wmf-config/CommonSettings.php ------------------------------**------------------------------**---------- 05:35 Tim: on db40: reduced to 10M, should be causing massive delays, but the site's not down and the purge rate is lower if anything. Going to disable the mysql parser cache entirely. ------------------------------**------------------------------**---------- 05:25 Tim: on db40: purge lag is still increasing at 108 per second, so reducing innodb_max_purge_lag to 50M ------------------------------**------------------------------**---------- 05:21 Tim: on db40: giving the innodb manual the benefit of the doubt and following its advice, setting innodb_max_purge_lag to 100M, which should give a delay of 4.5ms ------------------------------**------------------------------**---------- 05:13 Tim: killing purgeParserCache.php since it is probably doing more harm than good ------------------------------**------------------------------**---------- 02:43 maplebed: deployed updated thumb_handler.php to ms5 to include Content-Length in generated images ------------------------------**------------------------------**---------- 02:34 logmsgbot: LocalisationUpdate completed (1.19) at Fri Feb 17 02:34:32 UTC 2012 ------------------------------**------------------------------**---------- https://wikitech.wikimedia.**org/view/Server_admin_loghttps://wikitech.wikimedia.org/view/Server_admin_log
" Going to disable the mysql parser cache entirely."
Seems to have been reenabled on Feb 29th at 00:20:
00:20 Tim: reimported schema files on db40 and re-enabled mysql parser cache
-- Antoine "hashar" Musso
______________________________**_________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/**mailman/listinfo/wikitech-lhttps://lists.wikimedia.org/mailman/listinfo/wikitech-l