On May 9, 2005, at 10:47 AM, Tim Starling wrote:
One important thing to note is that the parser is a performance
problem,
not a scalability problem. We really can throw more hardware at it,
and
that will fix it quite nicely. In my opinion, our biggest performance
problem at the moment is not the parser, it's replication lag and
database load balancing. There's lots of work that has to be done
there.
Please excuse my ignorance, but why havn't different languages been
split up amongst different database servers?
let me explain what I mean in case that wasn't clear.
Lets say you have 2 Master database servers (for simplicity of
explanation) which replicate to 9 slaves which in turn answer to the
apaches. the apaches are not tied to any database cluster (these were
just split for a ~500k article threshold per cluster, once again just
for simplicity)
|-- S1.1 |
M1 - En |-- S1.2 | APACHES
|-- S1.3 |
|-- S2.1 |
M2 -De, Ja, fr, sv, pl |-- S2.2 | APACHES
|-- S2.3 |
This allows the Apaches to whore themselves around to whichever
cluster needs them at the moment. Ideally we could do a small traffic
study on the peak hours of each wikipedia and try to pair them off in
such a way that their peak hours miss each other allowing us to
maintain a relatively constant load o the servers (as one peak is
falling another one is rising). Somehow i feel that this would reduce
the loss from replication. Additionally, if there is data that is
shared amongst all wikipedias it could either be set aside in a
separate global cluster, or (is this even possible?) replicated to
all master servers.
What this would do is basically drastically reduce the amount of
replication data flowing from masters to servers, since we all know
that as the overhead of replication grows adding additional slaves
adds less and less additional capacity since most of the time the
slaves are just catching up on what happened on the master.
Now, I have not formally studied db design in school, but this seems
like a reasonable suggestion. Of course i am sure you guys thought
about this before and there is some reason it hasn't been mentioned.
What i would like to know is, if this has been proposed before, what
are the technical obstacles to implementing something like this?
Lightning