At 04:34 AM 1/17/2004, you wrote:
On Fri, 16 Jan 2004 17:50:33 -0700, Kelly Anderson
wrote:
This may be an old idea, or not. But the thought
of modularizing did bring
a thought to mind that is related to how Ebay does things...
What if the articles in a certain alphabetic range were stored on one
server, and those in another alphabetic range stored on another? A third
server stands in front doing nothing more than redirecting traffic to the
other servers. It might make statistics and such a bit more of a headache,
but it would certainly assist in reducing the bandwidth requirements of
the server.
Google does the same, i guess any really big DB application has to.
Well, someday, probably sooner than later, Wikipedia will be this type of
application, I'm guessing based upon it's growth rate.
Whether such
an implementation would be simplistic or terribly difficult
requires a greater understanding of the underlying architecture than I
have.
There would be a need to write logic to merge the separate result sets.
For example? I'm thinking simple, just the articles themselves... what
needs merging other than reports and change lists and things like that?
By the way, has any bench marking/profiling been done to determine which
functions are called most often, eat up the most server resources, etc.?
The temporary DB server is a very small machine with
something like 1Gb
ram. Geoffrey (with 4Gb) was fast before it went down, a second server is
ordered that can do reads in the new setup. I'd also expect the DB to be
the weakest spot in the new setup, but at a completely different level.
The number of DB connections should drop by 50% or so (even more with
memcached), the amount of ram will be sufficient to hold most of the DB in
ram, with the option to top it up to 16Gb on each machine.
--
Gabriel Wicke
Gabriel, it would seem that splitting up the database using vertical
partitioning (I think that's the right direction) across the alphabet based
upon the letter(s) the articles start with is one way of scaling. It is
clearly not the only way...
-Kelly