Gabriel Wicke wrote:
How about installing Squid on one of the machines? That would take a fair amount of load away. Is there a machine with some free Ram available? Even installing Squid on larousse would do i guess. I've glanced over the php code- there are mainly two header lines we would need to change to activate this- we could start off with a 30 minute timeout for anonymous users. Purging should get ready soon as well.
Perhaps I will be burned at the stake as a heretic for this, but I am not convinced squid proxies are the answer.
The delays in the wikiserver system are caused by waiting for I/O- the time taken for mechanical devices to seek a particular block of data. If the data is being served from a squid cache rather than from a cache on the wiki server, how will this reduce the overall I/O blocking problem? The busiest page data won't substantially add to I/O blocking on the wiki server as it will likely be in memory all the time. The squid proxy is ideal to solve the problem of network load from commonly accessed pages or pages which demand a lot of CPU power to generate but this is not a problem on wikipedia. If Squid proxies are being implemented to increase performance, then they are the right solution to the wrong problem. If they are to increase reliability by adding redundancy - multiple data sources-, they do this to a degree but are far from ideal.
The most commonly used pages are going to be in the memory of the database server so these are not costly to serve. The costly pages to serve are those which need disk seeks to serve. The more I/O seek operations a page requires, the more costly it is to serve.
The proxy server will need to make a database lookup (for the URL) and, unless the page is in memory rather than on-disk storage, use I/O to reach the fine grained data. The data for each unique URL will be bigger than that held in cache on the database server as it will contain html formatting and other page data. The likelihood of the data being in the memory of a proxy server is lower than the data being in memory of a similarly equipped database server as the data size of the final HTML page will be ~7.5k bigger than that of the database data.
If performance is the criteria, I suggest a proxy isn't a good idea. Instead, the memory otherwise used in a proxy would be better utilised caching database data directly. Either as a ramdisk or perhaps a network attached database storage with plenty of solid statememory.
From what I have gathered, the cost (limiting factor to performance) is that of delays seeking fine grained data. Either this seek load will need to be spread across many mechanical devices such that the work is not unduly duplicated, or store the fine grained data in solid state storage so that it can be seeked quickly.