Oh, *that* kind of profiling. I have done that
with C++ applications
under Windows, but I never thought it'd be necessary for webserver
applications... Clearly, at least our *current* problem - and also the
[...] is database performance, not CPU usage.
For ten years now, I've been working with performance issues in
systems that use databases, and I've never found it to be so easy as
you want it to appear.
Believing that "the problem is database performance" all too often
leads people to blindly buy new hardware or change database software
(from Oracle to Sybase or from MySQL to Postgres), as if printing more
money would solve any "shortage of money" sort of problem. Instead, I
think you have to look at *how* you *spend* the resources that you
have got, and consider if you could spend them otherwise. If,
perhaps, you are wasting and could start to save. As any economist
can tell you, the first step is monitoring the current flow, to get an
overview of what is going on. This is what the profiling is about.
It's not about CPU performance or database performance. It is about
locating the hole in the bucket.
The resource that we've got to work with is the two to five seconds of
patience that each user brings to www.wikipedia.org
. Before that time
runs out, they should have their content delivered. If we have a hole
in our bucket, where all time runs out, we better plug that hole.
If we make a single call to a database that takes seven seconds to
respond, we better fix that call (or the database). But if our code
for every HTTP request makes seventy calls to a database that always
responds in .1 seconds (if you think this example is unrealistic,
you haven't been in the telecom industry), the entire HTTP transaction
also takes a total of seven seconds, and we better fix our code to
make fewer calls rather than put the blame on the poor database.
It's about wallclock time, not CPU time. And profiling is the way to
go, if we want to do better than guess and pray. So far, the
Wikipedia developers have mostly been guessing and praying.
I think the worst performance I ever saw from Wikipedia was in May
2002. You can dig up the thread from the archives of this list, e.g.
Right now, Wikipedia is flying and everybody is happy.
Very true. I've been looking at the code, trying to find the db calls
where I could optimize something. and trying to find bottle necks, but
its pretty hard. The current code is *really* confusing. variables are
often passed as globals rather than parameters which makes it horrible
to look through the code and find whats going on. However, if anyone
knows of any specific place where you get the feeling that smething is
beeing done in a suboptimal place, name the action and I'll try to
follow through and dig up the whats going on.
I sent a proposal on a way to bring Special:Longpages and
Special:Shortpages back, but nobody replied. The code is very basic, and
it was actually partially included in my proposal, but I recieved no
reasons why it would be a good idea, or bad idea. I'd love to hear
*something* about it from someone who knows db's and if my proposal is
feasible or just unneccessary clunk.
Another fun think would be to log every query made to the test server
for say.. 500 page views or so along with a microtimestamp from the
begining to the end of the file. We could then properly analyze that
file and try to find the bottle necks and problematic query's. That
would be a great way to start optimization, by *finding* the problem
areas. Another nice stat to have would be a ratio of views to edits and
total views to watch list views. That would help us find ou where the
change would make the most difference.