On Mon, 12 Jan 2004 11:39:59 -0800 Brion Vibber brion@pobox.com wrote:

Well, went ahead and turned on interrupt unmasking and 32-bit io. The numbers in vmstat don't look too much different yet, but we'll see.

-- brion vibber (brion @ pobox.com)

On my IBM 80 Go, 7200 trs/mn

Before : IO_support = 0 (default 16-bit)

hdparm -t /dev/hdd /dev/hdd: Timing buffered disk reads: 82 MB in 3.05 seconds = 26.89 MB/sec

After : IO_support = 1 (32-bit)

hdparm -t /dev/hdd /dev/hdd: Timing buffered disk reads: 74 MB in 3.03 seconds = 24.42 MB/sec

No changes seen....

Shaihulud

On Mon, 12 Jan 2004, Nick Reinking wrote:

It seems to me that one of the biggest problems we seem to have is reliable and speedy hard drives. Perhaps it might be wise to consider possible purchasing an external disk subsystem? Something like the Apple Xserve RAID systems are speedy (lots of internal hardware RAID), reliable (easy to swap out disks), and expandable (up to 3.5TB, 1TB in the cheapest configuration). If the database server dies, it should be fairly easy to plug the external disk system into another machine.

"Like" is the key word there. 6k$ for a [censored] Apple logo is insane. I've bought similar hardware for less than 1/10th that price. If all you need is a drive shelf, start searching eBay. (I can recommendations if anyone cares.)

If you grab a fibre channel shelf (or more than one), I have plenty of drives for the cause (14x18G and 10x9.1G drives gathering dust.) I'd offer an entire Eurologic shelf, but you can find those local in CA easier and faster than shipping across the country. (6 of mine came from Canada :-))

--Ricky

On Mon, 12 Jan 2004, Nick Reinking wrote:

Hmm... 4GB Compact Flash cards would be:

Too small (4GB is not enough space, we would need a couple hundred cards)

Wait a sec, those two hundreds cards total 800GB. Are you sure that wikipedia needs that kind of storage? I thought that current articles are a few gigs, and history a few tens of gigs.

Too expensive (4GB cards cost approximately about $1100, or around $330000 for enough space) Too slow (fast access, but only a 5MB/sec read/write) Limit read/writes (they would last about one week, and need to be tossed out)

I agree, Flash cards are not OK for our needs. A server with lots of RAM is much better. Geoffrin would be really optimal, if it worked :(

Someone mentioned $6K for Apple raid storage, but that price is not so high: here at work we have some kind of raid NAS (half a terabyte I think) that went for about $4K. So yeah, NAS can be expensive, a simple internal SCSI raid like our current setup is enough, and with regular backups it can be transferred to another machine in case of need (heck, that's what Brion did multiple times in the last weeks if I am not mistaken, and that's enough proof that it works).

Alfio

Here's a crazy thought: Is there a way to make a server identify itself as a *harddrive* on the SCSI bus? If so, we could take a machine with lots of RAM which does nothing else than holding most of the DB in cache (cheaper and faster than Compact Flash:-) and occasionally write stuff to its hard drives. Probably won't need much of a CPU, and maybe not even SCSI drives, just cheap IDE ones, as all it does is caching.

Then, we could plug that thing into the "real" DB server, which just sees a really fast HD.

Well, one can dream...

I think I remember reading about projects fairly similar to this many years ago. However, I don't think they went anywhere. I imagine people realized that if it costs $10k to build a generic machine that can handle these disks, and a huge number of hours to configure it and get it running, then you were probably better off buying a more reliable $15k external disk system.

-- Nick Reinking -- eschewing obfuscation since 1981 -- Minneapolis, MN

On Mon, 12 Jan 2004, Alfio Puglisi wrote:

Someone mentioned $6K for Apple raid storage, but that price is not so high: here at work we have some kind of raid NAS (half a terabyte I think) that went for about $4K. So yeah, NAS can be expensive, a simple internal

And it's certainly IDE based. You might have it connected to a SCSI card but the drives are the same unreliable IDE drives everyone throws away every year. SCSI hardware RAID is, strictly speaking, unnecessary. The two opterons in the box are far more powerful than any RAID card. Sure, hardware is nice, but not really necessary with good drives from the get-go (i.e. they're all SCSI and we don't need some magic from a hardware RAID card to make IDE performance acceptable.)

I currently have a $250 eurologic fibre channel shelf (FC7 or FC9) with dual power supplies and 7 146G seagate (300$ each) heating my apartment :-) The shelf is easy to find. Wiki would want new drives vs. the OEM pulls I'm using which doubles the cost of the drives. (they aren't easy to find used anyway.) That's just under 1TB. Each drive can stream ~60MB/s. Yes, EACH. (all the way to bus saturation: 2Gb/s)

--Ricky

PS: I have 6 more of those shelves loaded with 18G drives... old NetApp filer disks.

On Mon, 12 Jan 2004, Nick Reinking wrote:

I'm not saying you can't build a similar system for cheaper (possibly a couple thousand dollars cheaper). But, do we want to spend a lot of time building and debugging hardware?

Uhhh, we're already doing that for a brand new server.

Do we want to be responsible for paying for any unforeseen hardware problems, or do we want to have some kind of support contract with someone? Imagine how much this Penguin Computing system would be costing us in time and money if we had to pay for all the replacement parts (especially when we are unsure where the problem lies.)

It'd cost exactly what it's currently costing. Everything in that server is new and thus supported both by the people who sold it to us and the people who made it. A pair of drives from Maxtor purchased 1.5 years ago were replaced in 48hours from the time requested the RMA. (it usually takes a week.)

Building your own hardware is great when you're desperate to save a few bucks, but if we've got the money, we might as well spend it on making things as reliable as possible. I think we would be doing a disservice to all those who made donations otherwise.

This isn't "penny pinching". The cost savings is enormous. And you're using hardware someone else has burned-in. Of course, you have to has some trust in the source of the gear.

If you want spankin' new, go price a shelf from IBM, DEC/Compaq, or Dell. A quick look at Dell shows a SCSI shelf with 14 36G 10k drives, dual 600W power supplies, rails, cables, etc. for ~6k. (ETA 8days.)

--Ricky

On Mon, 12 Jan 2004, Ricky Beam wrote:

I currently have a $250 eurologic fibre channel shelf (FC7 or FC9) with dual power supplies and 7 146G seagate (300$ each) heating my apartment :-) The shelf is easy to find. Wiki would want new drives vs. the OEM pulls I'm using which doubles the cost of the drives. (they aren't easy to find used anyway.) That's just under 1TB. Each drive can stream ~60MB/s. Yes, EACH. (all the way to bus saturation: 2Gb/s)

Yes, if one needs throughput, IDE or SCSI will hardly matter, and the former will be cheaper. But it seems that our disk subsystem is hampered by latency - lots of small reads looking for DB indexes, articles, things that got pushed out from the cache, all of them from multiple concurrent threads. Short of a pure ramdisk, SCSI raid is still the leader in this deparments :)

Alfio

Nick Reinking wrote:

I'm not sure how running two copies of MySQL on a machine is going to be faster than running one. Plus, I don't think MySQL can even work in this configuration.

I am not saying running two copies per se will be faster. Just that running a copy from a database file in ramdisk will be much faster. One instance of Mysql whose data file is on the mechanical disk, one instance being a replication slave whose data is on the ram disk. Data served from the ram disk, not the hard disk.

I am almost certain multiple instances of MySQL will co-exist on the same machine. If not directly, then through a chrooted environment. Sockets can span chrooted environments using 'mount --bind'.

Jimmy Wales wrote:

I think others have said this, but I think you're behind the times when you talk about a "ramdisk". We're not running DOS here. Linux will already cache as much as it can in memory, and in a sensible and automatic way.

But realistically speaking, this is not a bottleneck worth trying to solve. When it's working, the db server is very fast, and already holds everything that it needs in memory, as far as I know.

I understood the server was I/O bound from a recent posting. Server maxed out with low CPU utilisation. The information you and I have differs. You are in a position to have better information on this than I. I certainly don't want to waste time solving a non-existent problem.

Do we have a bottleneck? If so, where is it and how do we know?

On Tue, 13 Jan 2004 13:29:51 +0000, Nick Hill wrote:

Do we have a bottleneck? If so, where is it and how do we know?

The temporary server doesn't have enough ram, that's all. Geoffrin has 4Gb and worked fine, adding more ram to Ursula would do the same.

On Tue, 13 Jan 2004 05:46:15 -0800, Jimmy Wales wrote:

I'm putting the finishing touches on an order at Silicon Mechanics right now, and I'll report back to everyone once I have all the details and a firm delivery date promise.

Jimbo-

how much ram are you going to order for the Squids? I've read that it's possible to address more than 4Gigs though highmem on x86, but only 3 Gigs per process. Squid is a single-thread asynchronous app, so this is a limit. Is there a major price penalty in a single-processor 64bit CPU? Having enough ram for all content on the Squids is a good thing for both speed and reliability- not an immediate problem for now, but it might get one as wikipedia grows. On the other hand x86 Squids could be converted to Apaches later if they are the same kind of machine.

The ram issue propably also applies to the DB server, but i guess you'll order 64bit anyway for that.

== Access patterns ==

I did some stats ages ago, showing an approximately Zipfian distribution for page accesses. A bit of calculation shows that for small numbers of articles, this means a small amount of cache will give a large performace boost. However, as the number of articles increases, the vast majority of seldom-accessed articles will start to dominate the behavior of the system for article fetches. Thus, RAM caching will decrease in usefulness over time as the project progresses, unless the RAM cache is close to the size of the entire working set.

Nick's suggestion of tuning the filing system page size to the article size is a good idea; it will tend to make the RAM cache which is currently available more effective. I'm rather dubious about some of his other suggestions.

Where RAM caching is really important is in the "hot" data such as article timestamps and link tables. These have already been partially addressed by the use of memcached, I believe. These commonly accessed pieces of data should be small enough to keep in RAM all the time, giving a large speedup to the system.

== Seek bound performance ==

Since disk I/O requests are effectively random, the load will be dominated by seek and rotational latency. It will cost very nearly the same to pick 64kbytes off the disk for an article as to get 4 bytes for a timestamp.

Using high-performance disks and spreading the database across many RAID spindles should greatly increase performance.

I agree with the posters who are arguing for software RAID: it has higher performance than hardware RAID in many cases, and again, we can fine-tune stripe sizes etc. to our application. (Big stripe sizes are a bad idea for random-seek loads, but give better performance for streaming loads). We should also consider kernel 2.6: there are major gains in disk I/O performance in this kernel, and most of the teething troubles are not related to server issues.

== Not all disks are equal ==

Consider buying the disks specifically by access time statistics. In particular, high-performance SCSI disks should greatly out-perform IDE for random seek access patterns, even though their performance may be roughly the same for data streaming. SCSI command tagging will further increase performance, where there is concurrency on a single spindle.

See http://www.storagereview.com/php/benchmark/bench_sort.php for some interesting stats:

* a Fujitsu MAS3735 has an average read access time of 5.6ms, for a price of $700 for 73 GB. * a Hitachi Deskstar 7K250 has an average read access time of 12.1ms, for a price of $250 for 250 GB * a seagate U6 has an average read access time of 20.0ms, for a price of ??? for 80 GB

According to this, if performance is dominated by read access time, the most expensive drive should have almost four times the random-read performance of the cheapest, all else being equal.

Using price and performance figures such as those above, we should be able to calculate the best price/performance/storage compromise for this application.

== The Google strategy for article caching ==

Google seem to use a large number of RAM-based cache servers, based on the observation that network access latency on a small network is tiny, but disk latency is large. This does not make any sense for us now: we don't have the resources, unless Google open-source their Google filesystem.

For future expansion, it might be cheaper to buy 10 4Gbyte RAM commodity machines than one 40 Gbyte enterprise-class machine, and spread the load across them. Although this would still be costly, the performance of serving data directly from RAM would be very high.

-- Neil */

/*

On Tue, 13 Jan 2004, Nick Reinking wrote:

Also, it should be noted that if we want to extend the usefulness of current machines, there are some 10K Western Digital IDE drives (basically SCSI drives with an IDE interface). They have an average read access time of 8.3ms (the WD740GD does, at least), and is $260 from Newegg for the same size (74GB). It also has a built-in PATA-to-SATA bridge, so it supports hot-swapping and command queueing.

Indeed. Those drives are "bastard stepchildren"... it's a SCSI servo with IDE controller electronics tied to a SATA bus via a built-in bridge. It's a f'ing IDE drive. Stay away from it. If you want SATA (read: something cheaper than SCSI), buy real, native SATA drives which only Seagate are currently making. (which exact models, I don't know.)

Haven't we already had the "cheap is not the goal" mini-flame? If you want fast and reliable, it's gonna take some green. If you want cheap, it'll be less reliable and slow. I would personally recommend a FC array deployed under LVM2 with XFS riding on top. The array is physically expandable to 127 drives. And the drives are really fast, esp. when you have 100 of them in parallel :-)

--Ricky

This is my understanding of how it works:

There are two RAID devices in each box. Each device supports up to 7 drives, and can use RAID 0, 1, 3, 5, 0+1, 10, 30, and 50 with these seven disks. Now, if you want to use the entire 14 in one gigantic RAID setup, you can do that with software RAID (you can do RAID 10, 30, or 50 across the two RAID systems).

This is not actually terribly optical. It is best to use the two RAID systems separately. If you want a gigantic file system, you can use LVM and pull both RAID systems in (probably running RAID 5). However, how I really think it would be most useful would be if we had two database servers. One would run en, and one would run the international wikis. Since the RAID system has two fibre channel ports, you can give each system its own external RAID hardware subsystem in the future. (so, both could have up to 1.75TB of storage at their disposal).

In the near term, just using half the machine (7 disks, just one RAID controller) would still make a huge improvement, I think. It's not the cheapest solution, but it is a lot more reliable than hoping the disks in the servers don't croak, and it gives us the ability to move the database around fairly easily if a server should die (just plug it into a different machine).

-- Nick Reinking -- eschewing obfuscation since 1981 -- Minneapolis, MN

On Jan 12, 2004, at 2:35 PM, audin@okb-1.org wrote:

As to the Apple RAID boxes, my investigations a while back indicated that they are not really redundant. Each box is two seperate RAID devices. So you have to do software raid between them if you want to avoid a single point of failure. This may or may not be a problem, but is not explicitly mentioned in the apple literature. I'm not sure this is a deal breaker, but it is certainly something to think about.

how about the cost?

--- Nick Hill nick@nickhill.co.uk wrote:

This would be cured by moving the database to solid state memory.

__________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus

On Mon, 12 Jan 2004, Gabriel Wicke wrote:

On Mon, 12 Jan 2004 11:11:28 -0800, Brion Vibber wrote:

...

If Geoffrin's not going to be up soon, and Pliny's still emitting spurious disk errors, what are our options out of the available machines?

How about installing Squid on one of the machines? That would take a fair amount of load away. Is there a machine with some free Ram available?

It seems a good idea. It would eliminate the DB timestamp lookup for each page request. But the purging code must be implemented, and that would take time.

Alfio

Gabriel Wicke wrote:

How about installing Squid on one of the machines? That would take a fair amount of load away. Is there a machine with some free Ram available? Even installing Squid on larousse would do i guess. I've glanced over the php code- there are mainly two header lines we would need to change to activate this- we could start off with a 30 minute timeout for anonymous users. Purging should get ready soon as well.

Perhaps I will be burned at the stake as a heretic for this, but I am not convinced squid proxies are the answer.

The delays in the wikiserver system are caused by waiting for I/O- the time taken for mechanical devices to seek a particular block of data. If the data is being served from a squid cache rather than from a cache on the wiki server, how will this reduce the overall I/O blocking problem? The busiest page data won't substantially add to I/O blocking on the wiki server as it will likely be in memory all the time. The squid proxy is ideal to solve the problem of network load from commonly accessed pages or pages which demand a lot of CPU power to generate but this is not a problem on wikipedia. If Squid proxies are being implemented to increase performance, then they are the right solution to the wrong problem. If they are to increase reliability by adding redundancy - multiple data sources-, they do this to a degree but are far from ideal.

The most commonly used pages are going to be in the memory of the database server so these are not costly to serve. The costly pages to serve are those which need disk seeks to serve. The more I/O seek operations a page requires, the more costly it is to serve.

The proxy server will need to make a database lookup (for the URL) and, unless the page is in memory rather than on-disk storage, use I/O to reach the fine grained data. The data for each unique URL will be bigger than that held in cache on the database server as it will contain html formatting and other page data. The likelihood of the data being in the memory of a proxy server is lower than the data being in memory of a similarly equipped database server as the data size of the final HTML page will be ~7.5k bigger than that of the database data.

If performance is the criteria, I suggest a proxy isn't a good idea. Instead, the memory otherwise used in a proxy would be better utilised caching database data directly. Either as a ramdisk or perhaps a network attached database storage with plenty of solid statememory.

From what I have gathered, the cost (limiting factor to performance) is that of delays seeking fine grained data. Either this seek load will need to be spread across many mechanical devices such that the work is not unduly duplicated, or store the fine grained data in solid state storage so that it can be seeked quickly.

On Mon, 12 Jan 2004 16:23:30 -0800, Brion Vibber wrote:

The ideas about squid caches etc are not about lightening the load on the database server

Ahem.

(which it wouldn't really do except insofar as it may cache more stuff than our present on-web-server caching), but about lightening and spreading out the load on the web servers. Squid caches will *not* help the immediate question here to a significant degree; anyway no more than making slight alterations to the present caching code to avoid checking timestamps in some cases would.

Our present alternatives for database duty are:

• Pliny, which has done it in the past. Exhibiting intermittent errors

on primary drive, and crashed a couple times when it ran the database again in late December, which is why we took it back off to Ursula.

• Carol (currently idle) with a SCSI drive that's too small for the

whole database.

Doesn't matter, that's what a replacement algo is for. There are much more sophisticated ones than LRU to pick from- both for ram and for disk.

• Susan (currently idle) with another IDE drive that's likely not the

fastest.

If these two servers have some Ram they'll be fine. We could even disallow disk caching all together, only using their Ram.

On Mon, 12 Jan 2004 20:16:27 -0800, Brion Vibber wrote:

Scratch Carol off the list for now. Carol's also got 4GB of RAM which would be _very_ nice to have up and running, assuming it all works!

Are there unused cache slot on Ursula? If we could use some of this ram we should be set i guess- Geoffrin also has 4 Gigs and didn't seem to swap badly. Might be imcompatible ram though, don't know.

...

• Susan (currently idle) with another IDE drive that's likely not the

fastest.

Susan hasn't died during memory testing yet (yay). However it's a smaller machine with only 512MB of memory as well as the ide disk. Not ideal for database, though this might be sufficient to offload some work.

Hm- could we stick some of Carol's ram into this one?

On Jan 12, 2004, at 17:01, Gabriel Wicke wrote:

If we have an old machine Squid will serve anything cached straight from memory (small objects) or disk (images) without ever contacting the database. That's a speedup of at least 50x over the current disk cache with DB lookup etc.

...which would only work as such if we make changes to the wiki which are roughly identical to the changes we'd have to make to the present caching system to avoid hitting the database to confirm timestamps. (ie, explicit purging of cached pages that are no longer valid)

Squid may be a great caching system, but for purposes of this discussion it's not really different from what we've got.

-- brion vibber (brion @ pobox.com)

On Jan 12, 2004, at 17:48, Gabriel Wicke wrote:

On Mon, 12 Jan 2004 17:15:38 -0800, Brion Vibber wrote:

...

Squid may be a great caching system, but for purposes of this discussion it's not really different from what we've got.

Believe me that Squid is *way* faster than any php caching system will ever be- with modification date checking or not.

This thread is about disk access on the temporary database server.

No matter how insanely fast the caching is, it doesn't happen on the database server. Superduperfast squid caching that doesn't touch the database server and kinda speedy php caching that doesn't touch the database server are exactly equal in the eyes of the database server.

Lemme put it this way: you're talking about tuning up the engine on a car, when we're trying to discuss how the tires all are flat and the parking brake is stuck on. The car's not going to go very fast in this condition no matter how much the engine is improved.

-- brion vibber (brion @ pobox.com)

Gabriel Wicke wrote:

On Tue, 13 Jan 2004 00:07:50 +0000, Nick Hill wrote:

...

The most commonly used pages are going to be in the memory of the database server so these are not costly to serve. The costly pages to serve are those which need disk seeks to serve. The more I/O seek operations a page requires, the more costly it is to serve.

Yup. So lets avoid them.

Given that popular articles will be in the database memory cache, requests for popular articles should not lead to database HDD seeking.

I would expect a Squid proxy be best at serving popular pages and poor at serving less popular pages. So I can't imagine how squid is very helpful at saving HDD seeks.

...

The proxy server will need to make a database lookup (for the URL)

Nope. Only if a page is *not* in the cache or marked as not cacheable.

I meant the squid server will need to look up it's own database (in whatever form that may be- filesystem or indexed DBMS) to check if it has a copy of the required data- using the URL as a key. If it has a copy of the data, it will need to either pull it out of memory or from the disk. If not, forward the request then add the page to it's own database.

If the squid server needs to pull an article page off the disk, then disk I/O will be required in the same way as it would be required if an uncached piece of data is read from the database by the web server.

As I/O is the bottleneck, the squid server is likely to suffer the same problems as the underlying database server. These problems are likely to be bigger as the chunks of fine grained data handled by squid will be larger (compressed fully formed HTML pages) than the fine grained chunks of data (article text) handled by the database server.

...

If performance is the criteria, I suggest a proxy isn't a good idea.

Well- please read up some docs. Or benchmark http://www.aulinx.de/ - commodity server (Celeron 2Ghz) running Squid.

I am not contending that squid is not a very high performance server. I believe it is a high performance server. I believe it can substantially reduce the bandwidth needed by ISPs to serve web surfers.

The issue for wikipedia is how many disk accesses, in total, are needed for each article hit.

Wikipedia has millions of discrete pieces of data. Most pieces referenced singularly by a unique URL. Squid will not be able to hold a substantial proportion of these in memory. Squid will be able to hold fewer of these data chunks (articles/ HTML pages) in a given amount of memory than a database server could as the articles stored on the database consist of less data than the article rendered in HTML. For the larger articles, the comressed html page will be smaller, but for most articles, the compressed HTML page will be bigger. (the relative weight of the page HTML is much greater for short articles than for long articles. Compression reduces page size by about half.)

I assume viewing an article history page requires several pieces of information leading to multiple seeks per request: If squid were able to serve article histories, then a single I/O on the squid box could save several database seeks on the database server, providing a substantial economy. However, individual page histories are each fairly rare and forcing a squid cache reload on an article history page when an article is updated may be a poor use of resources.

I suggest four avenues for investigation: 1) Store articles in the MySQL table in compressed (gzip) format. This will reduce the size of the articles, making them fit more easily into the available cache memory, increasing the chances of a cache hit almost by a factor of two. Perhaps this can be made as a patch to MySQL. 2) Investigate ways of prioritising data cached in memory such that smaller chunks of data have a higher value than larger chunks so that smaller chunks are not flushed according to the basic least recently used algorithm. To reflect the relative cost of reading a small chunk of data from the HDD. 3) If the SQL code underlying wikipedia relies on temporary tables as part of the SQL queries, investigate whether the I/O of writing temporary tables tends to flush data from the disk cache. If so, write temporary tables to ramdisk or other storage which does not cause flushing. More recent versions of MySQL support sub-queries. This may obviate the need for temporary tables. 4) Judicious use of solid state storage. Could dramatically reduce seek times and I/O bottleneck. Some issues to resolve regarding flash memory durability and possible MySQL hotspots. Also cost of mass solid state storage. Might be worthwhile for some wiki data.

At 00:07 13/01/2004 +0000, you wrote:

Gabriel Wicke wrote:

...

How about installing Squid on one of the machines? That would take a fair amount of load away. Is there a machine with some free Ram available? Even installing Squid on larousse would do i guess. I've glanced over the php code- there are mainly two header lines we would need to change to activate this- we could start off with a 30 minute timeout for anonymous users. Purging should get ready soon as well.

Perhaps I will be burned at the stake as a heretic for this, but I am not convinced squid proxies are the answer.

You should not be burnt

The delays in the wikiserver system are caused by waiting for I/O- the time taken for mechanical devices to seek a particular block of data. If the data is being served from a squid cache rather than from a cache on the wiki server, how will this reduce the overall I/O blocking problem?

Agreed

The busiest page data won't substantially add to I/O blocking on the wiki server as it will likely be in memory all the time. The squid proxy is ideal to solve the problem of network load from commonly accessed pages or pages which demand a lot of CPU power to generate but this is not a problem on wikipedia. If Squid proxies are being implemented to increase performance, then they are the right solution to the wrong problem. If they are to increase reliability by adding redundancy - multiple data sources-, they do this to a degree but are far from ideal.

The most commonly used pages are going to be in the memory of the database server so these are not costly to serve. The costly pages to serve are those which need disk seeks to serve. The more I/O seek operations a page requires, the more costly it is to serve.

The proxy server will need to make a database lookup (for the URL) and, unless the page is in memory rather than on-disk storage, use I/O to reach the fine grained data. The data for each unique URL will be bigger than that held in cache on the database server as it will contain html formatting and other page data. The likelihood of the data being in the memory of a proxy server is lower than the data being in memory of a similarly equipped database server as the data size of the final HTML page will be ~7.5k bigger than that of the database data.

One solution is to lower the options available to the user to make the pages more static (unpopular)

If performance is the criteria, I suggest a proxy isn't a good idea. Instead, the memory otherwise used in a proxy would be better utilised caching database data directly. Either as a ramdisk or perhaps a network attached database storage with plenty of solid statememory.

Probably the only solution apart from splitting the various wikis to more servers

From what I have gathered, the cost (limiting factor to performance) is that of delays seeking fine grained data. Either this seek load will need to be spread across many mechanical devices such that the work is not unduly duplicated, or store the fine grained data in solid state storage so that it can be seeked quickly.

Or data replication over more/many db servers.

We really do need to spread the load (high power processors are less important than high speed disk.

As things are at the moment to cache the db to get full performance we need a server with 50gig ram (we cannot get 4gig working yet).

So for the next best thing I believe the suggested hot swap disk array is needed for both performance and reliability (2 off) to make two fast db servers.

plus squids apaches and dns round robin etc

Dave Caroline aka archivist