Um, there is, I, (pause to remove foot from mouth)
Reducing time from .99 to .97 doesn't sound very important. But I still think that efforts to reduce page delivery time are important. Any idea, no matter how far-fetched, should be considered.
Naturally, the proof is in the pudding. So after we try a given optimization, we ought to measure the results and see if it really gets the pages to the users any faster.
Ed "Sheepish" Poor
On Wed, Feb 25, 2004 at 11:43:18AM -0500, Poor, Edmund W wrote:
Um, there is, I, (pause to remove foot from mouth)
Reducing time from .99 to .97 doesn't sound very important. But I still think that efforts to reduce page delivery time are important. Any idea, no matter how far-fetched, should be considered.
Naturally, the proof is in the pudding. So after we try a given optimization, we ought to measure the results and see if it really gets the pages to the users any faster.
Ed "Sheepish" Poor
Sorry to be a balloon burster. ;) Do we have any recent profiling on the code that shows where the bottleneck is? I imagine most of the time is spent poking about in the database. If that truly is the case, developer time would best be spent trying to maybe optimize the database layout or adding support for a faster database...
On Wed, 25 Feb 2004, Nick Reinking wrote:
Sorry to be a balloon burster. ;) Do we have any recent profiling on the code that shows where the bottleneck is?
E23 allegedly has been working on profiling code, but I've not played with it in a long time so can't say what results look like.
Last time we did profiling, the Prime #1 Enemy was OutputPage::replaceInternalLinks(). The rest of the page parsing was zippy fast even with a hojillion passes.
This requires splitting up the page, identifying links, normalizing titles, checking lookup tables for page existence, and generation of output links. Make it fast and you win a prize!
I imagine most of the time is spent poking about in the database. If that truly is the case, developer time would best be spent trying to maybe optimize the database layout or adding support for a faster database...
That's what the next revision will concentrate on: cleaning up the schema, moving the large data blobs out of the most-queried tables, etc.
Note that page parsing time is largely separate from (though can be slowed down by locking or disk churning due to) database slowness. But an overloaded database does slow everything down and makes people very unhappy.
-- brion vibber (brion @ pobox.com)
On Wed, Feb 25, 2004 at 05:17:02PM +0000, Brion Vibber wrote:
Last time we did profiling, the Prime #1 Enemy was OutputPage::replaceInternalLinks(). The rest of the page parsing was zippy fast even with a hojillion passes.
This requires splitting up the page, identifying links, normalizing titles, checking lookup tables for page existence, and generation of output links. Make it fast and you win a prize!
Depending on how much time replaceInternalLinks spends doing regular expression matching (as opposed to looking up page existence), you could use the Flex code I have to nearly completely optimize away link formatting. It would be very easy to update the code to convert the link into a singular format - that way, there would be no need to do the PHP regular expression matchings.
Kind of a pity that PHP doesn't let you compile regular expressions. :(
Brion Vibber wrote:
On Wed, 25 Feb 2004, Nick Reinking wrote:
Sorry to be a balloon burster. ;) Do we have any recent profiling on the code that shows where the bottleneck is?
Last time we did profiling, the Prime #1 Enemy was OutputPage::replaceInternalLinks(). The rest of the page parsing was zippy fast even with a hojillion passes.
This requires splitting up the page, identifying links, normalizing titles, checking lookup tables for page existence, and generation of output links. Make it fast and you win a prize!
My guess is that the slowest part of it is checking whether a page exists, and if it does, checking its size (if the user has set the preference that shows stubs in a different colour), because both of this requires a database query.
These two things are therefore perfect candidates for memcaching (hint, hint). Notice that we need to check if a page exists in several other places too (hint, hint).
Timwi
On Feb 25, 2004, at 14:48, Timwi wrote:
My guess is
I hereby forbid all discussion of optimization and slow spots until somebody profiles actual running code and has data to show where the slow spots are.
-- brion vibber (brion @ pobox.com)
Brion Vibber wrote:
On Feb 25, 2004, at 14:48, Timwi wrote:
My guess is
I hereby forbid all discussion of optimization and slow spots until somebody profiles actual running code and has data to show where the slow spots are.
Seconded.
In my opinion, performance is perfectly fine right now. It could be better, of course, I mean, anything could always be better, right?
And we're running with a suboptimal db server. suda has a load average of 3-4 fairly often, while geoffrin (when running) had a load average under 1.0 m ost of the time.
When Geoffrin is healed and here (just over a week from now, I think), then we'll have 2 db servers, and we can do some simple load balancing on those.
And our current architecture is now very reasonably scalable, *after* we come to some understanding of where bottlenecks might be. If we need more webservers, or more RAM in some squids, or more database servers, we will find it fairly easy to just add them.
I'm personally still a lot more focussed on *reliability*. We want to work hard to make sure that wikipedia always answers every request.
I've already made my campaign against worrying about arcane possibilities like datacenter failure and whatnot. Those are good to think about in the abstract, but we have other issues *right now*.
(Although nothing so serious as what we faced over the holidays, of course.)
--Jimbo
Nick Reinking wrote:
I imagine most of the time is spent poking about in the database.
I totally agree.
If that truly is the case, developer time would best be spent trying to maybe optimize the database layout or adding support for a faster database...
...or memcache more. :-)
Timwi
Poor, Edmund W wrote:
Um, there is, I, (pause to remove foot from mouth)
Reducing time from .99 to .97 doesn't sound very important. But I still think that efforts to reduce page delivery time are important. Any idea, no matter how far-fetched, should be considered.
Naturally, the proof is in the pudding. So after we try a given optimization, we ought to measure the results and see if it really gets the pages to the users any faster.
Ed "Sheepish" Poor _______________________________________________ Wikitech-l mailing list Wikitech-l@Wikipedia.org http://mail.wikipedia.org/mailman/listinfo/wikitech-l
A dumb suggestion:
Temporarily, set the code to parse each page _twice_ (or N times) instead of once, and see if it makes things any _slower_, and if so, how much.
This should give you a good way of estimating the current slowdown, without writing much code. Then you can estimate the possible speedup, and whether it's worth the effort.
-- Neil
Neil Harris wrote:
A dumb suggestion:
Temporarily, set the code to parse each page _twice_ (or N times) instead of once, and see if it makes things any _slower_, and if so, how much.
This should give you a good way of estimating the current slowdown, without writing much code. Then you can estimate the possible speedup, and whether it's worth the effort.
This isn't actually quite so bad an idea; this could be done for about a day to gather the necessary data. However, it *only* tells us how much of the performance is eaten up by the parser. It doesn't really tell us anything else. Our biggest problem is most likely not the parser.
Timwi
Wikimedia developers wikitech-l@Wikipedia.org writes:
This isn't actually quite so bad an idea; this could be done for about a day to gather the necessary data. However, it *only* tells us how much of the performance is eaten up by the parser. It doesn't really tell us anything else. Our biggest problem is most likely not the parser.
To test if resolving internal links is a big performance hit, create two pages, of roughly the same size, one with a large number of internal links, and one with a large number of external links, which require no database work.
You might want to create another page with *undefined* internal links, to test if there is some strange database problem.
Can you time processing of an individual page rendering?
Nick Pisarro
wikitech-l@lists.wikimedia.org