On 14/03/11 12:37, Daniel Friesen wrote: Yes, regular page views caused the load, after the cache was invalidated. After the jobs were run. The edits were made over a week ago. It affected the whole cluster. I thought I already said that. Secure is just a single server which acts as a proxy. It decrypts the requests and forwards them back to the main apache cluster. There was no problem with secure, it was reporting errors appropriately. For now I have added a sleep() to the code to limit invalidations to 100 pages per second per job runner process. -- Tim Starling

On 14/03/11 19:52, Domas Mituzas wrote: That's why Platonides wrote it for us. Now we just have to package it and deploy it. There was a CPU spike after my change, but that was probably because 100 invalidations per second was still too fast, rather than a Michael Jackson type of problem. On IRC, Daniel Friesen suggested pushing new HTML into the parser cache from the job queue instead of invalidating. Maybe after we move the parser cache to disk, that will be a good option. -- Tim Starling

On 15/03/11 02:44, William Allen Simpson wrote: Well, an "MJ issue" is a special kind of performance problem that affects very popular pages, like how popular the [[Michael Jackson]] article was in the few hours after his death. See wikitech-l from around that date. The "long tail" is a buzzword referring to the low-frequency end of a Zipf distribution, i.e. the least popular articles. So it's hard to see how an MJ issue could be in the long tail. I think he just means an MJ issue. -- Tim Starling

On 3/14/11 6:34 PM, Daniel Friesen wrote: Sorry, you folks get used to communication via IRC, and then use IRC'ish internal speak.... Anyway, I remember the MJ day, and went back and read that thread. Whatever happened to Domas' idea that, for things already in the cache and about to be invalidated, a newly parsed instance be stored in memcache before invalidating other caches, so that the article is only parsed once?

On 11-03-14 05:14 AM, Tim Starling wrote: To avoid pushing frequently used entries out, I also thought of the idea of only replacing caches that exist. ie: If it was parser cached, replace the parser cache with a new version, if not, leave it alone so we don't flush frequently used entries with infrequently used ones. Unless cache replacements still flush a lot of frequently used entries that option might have most of the advantages of updating caches from the job queue while avoiding the biggest pitfalls. There is also the option of trying to distribute these timewise and break them up into batches so there is time for normal operations to re-cache without doing it all at once. Though that project might be a little more involved since it'll depart from the normal operation of the job queue and have jobs that shouldn't be run off the queue right away. I did always love the idea of a job daemon though. ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]