In the last week, my script made 410 attempts at 20 minute intervals to reach the page http://www.wikipedia.com/wiki/Chemistry Out of these, only 86% were served in less than 5 seconds. Five percent of the calls timed out (my limit is 60 seconds). Now, this is far better than the worst problems that Wikipedia saw in April or May, but it is still pretty miserable.
I believe the performance in the new code is much improved.
Even under all the load I can put on it with a T1 line, the beta software is producing average load times of less than 1.5 seconds.
A number of people are now working on stress testing this software before it is put into production. And I think there is a general commitment to solving the performance problem, and I see lots of movement in the right direction.
On Sat, 13 Jul 2002, Mark Christensen wrote:
A number of people are now working on stress testing this software before it is put into production. And I think there is a general commitment to solving the performance problem, and I see lots of movement in the right direction.
This is fine, but you are trying a new car, and I want you to mount a speedometer so we can monitor the speed while driving, continuously keeping an eye on the problem instead of making believe we can fix it once and for all. The timer that is included in an HTML comment could be appended to a log file, and then there would have to be a way to review that log file. We did discuss this before, and someone was guessing that appending a line to a file would take too much time, but I don't think anybody tested this. The append to file could be conditioned so only accesses that took more than 2 (or 5) seconds were logged.
And, needless to say, this needs to be done on all web pages, including the edits, diffs, and specials, and for all languages.
Lars Aronsson wrote:
We did discuss this before, and someone was guessing that appending a line to a file would take too much time, but I don't think anybody tested this. The append to file could be conditioned so only accesses that took more than 2 (or 5) seconds were logged.
In my opinion, this would work just fine. At bomis, which is run on fastcgi/perl scripts cruder than you can possibly imagine, I frequently hand-tune things by writing logfiles in this fashion. At least in perl/fastcgi, appending loglines to a file is fast enough that it plays only the very tiniest role in the overall performance of the site.
It's a very useful tool for getting a mental picture of what's going on.
--Jimbo
Jimmy Wales wrote:
Lars Aronsson wrote:
We did discuss this before, and someone was guessing that appending a line to a file would take too much time, but I don't think anybody tested this. The append to file could be conditioned so only accesses that took more than 2 (or 5) seconds were logged.
In my opinion, this would work just fine. At bomis, which is run on fastcgi/perl scripts cruder than you can possibly imagine, I frequently hand-tune things by writing logfiles in this fashion. At least in perl/fastcgi, appending loglines to a file is fast enough that it plays only the very tiniest role in the overall performance of the site.
It's a very useful tool for getting a mental picture of what's going on.
--Jimbo _______________________________________________ Wikitech-l mailing list Wikitech-l@ross.bomis.com http://ross.bomis.com/mailman/listinfo/wikitech-l
I agree: logfile writes will normally be buffered in disk cache RAM by the OS unless a file is explicitly sync'd, so the overhead is perhaps one disk hit per 5 seconds, at an average cost of a seek, half a spin, and a single block write: perhaps 15 ms in 5s, or 0.3% of total disk performance.
Using Apache's log rotation system is good too: having, say, 14 logs and rotating them daily will keep two weeks of logs. At 100,000 hits/day and 80 chars/log entry, each logfile is 8 Mbytes, and the entire set of logs will only take 112 Mbytes on disk: hardly noticeable on modern systems with tens of gigabytes of spare disk space.
Neil
wikitech-l@lists.wikimedia.org