On 11-03-14 05:14 AM, Tim Starling wrote:
On 14/03/11 19:52, Domas Mituzas wrote:
For now I
have added a sleep() to the code to limit invalidations to
100 pages per second per job runner process.
Mass invalidations will create MJ
issue in the long tail though... Need poolcounter ;-)
That's why Platonides
wrote it for us. Now we just have to package it
and deploy it.
There was a CPU spike after my change, but that was probably because
100 invalidations per second was still too fast, rather than a Michael
Jackson type of problem.
On IRC, Daniel Friesen suggested pushing new HTML into the parser
cache from the job queue instead of invalidating. Maybe after we move
the parser cache to disk, that will be a good option.
-- Tim Starling
To avoid pushing frequently used entries out, I also thought of the
idea
of only replacing caches that exist.
ie: If it was parser cached, replace the parser cache with a new
version, if not, leave it alone so we don't flush frequently used
entries with infrequently used ones.
Unless cache replacements still flush a lot of frequently used entries
that option might have most of the advantages of updating caches from
the job queue while avoiding the biggest pitfalls.
There is also the option of trying to distribute these timewise and
break them up into batches so there is time for normal operations to
re-cache without doing it all at once. Though that project might be a
little more involved since it'll depart from the normal operation of the
job queue and have jobs that shouldn't be run off the queue right away.
I did always love the idea of a job daemon though.
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [
http://daniel.friesen.name]