On Jan 3, 2004, at 13:46, Gabriel Wicke wrote: Every page view comes in through wiki.phtml as the entry point. This runs some setup code, defines functions/classes etc, connects to the database, normalizes the page name that's been given, and checks if a login session is active, loading user data if so. Then the database is queried to see if the page exists and whether it's a redirect, and to get the last-touched timestamp. If the client sent an If-Modified-Since header, we compare the given time against the last-touched timestamp (which is updated for cases where link rendering would change as well as direct edits). If it hasn't changed, we return a '304 Not Modified' code. This covers about 10% of page views. If it's not a redirect, we're not looking at an old revision, diff, or "printable view", and we're not logged in, the file cache kicks in. This covers some 60% of page views. If saved HTML output is found for this page, it's date is checked. If it's still valid, the file is dumped out and the script exits. The cache file is a complete gzipped HTML page; if the browser doesn't advertise understanding gzip, we decompress it on the fly. (Note that this may affect benchmarks in comparison to actual browsers in use, I don't know.) If the cached page doesn't exist or is out of date, page rendering continues as normally, and the output is compressed and saved at the end. About 2% of page views involve saving a new cached page. There's no timeout; pages are invalidated immediately by updating their last-touched timestamps. A global cache epoch can be set on the server to invalidate all old cached pages (server- or client-side), and individual user accounts also have a cache epoch which is reset on login, when user options are changed, and when talk page notification comes on/off. If this is a redirect, old page view, diff, or printable view, or if the user is logged in, then we don't do any server-side caching (yet) and parse/render the whole page. Some speedups have been accomplished by precaching link lookup info in easily-loadable chunks. E23's been working on storage of the HTML-rendered wiki pages to be inserted into the overall layout, but this needs some more finalization (various user options may affect the rendering of the page). Ideally we'd be putting cached data into memcached, which can run in-memory on the web server (or as a distributed cache over a web server cluster) without grinding down the disks. So far we use memcached just for some common data (localized messages, utf8 translation tables, interwiki prefix lookup) and login sessions. -- brion vibber (brion @ pobox.com)

On Jan 3, 2004, at 17:33, Gabriel Wicke wrote: In theory, yeah, that should rock. :) I seem to recall some trouble with Vary headers other than "Vary: User-Agent" and Internet Explorer. This can probably be worked around in some way, I hope. There's no real good reason for it, just laziness. Session tracking isn't actually used outside of a login session, so it's just annoying for people who browse in with their browser set to prompt about cookies. In theory, we should be able to switch on the session system only after either a) checking that a session cookie is present or b) establishing a login session on Special:Userlogin. Note however that we want to be able to display talk page notifications to individual anonymous users, as well; currently this requires a hit to the server to check if the IP is marked. It may be sufficient to start up the cookie session on edit; most casual visitors may never get there, and talk notifications aren't too likely to happen to somebody who's not editing at the time. -- brion vibber (brion @ pobox.com)

On Sun, 04 Jan 2004 04:08:45 +0100, Gabriel Wicke wrote: Obviously time to go to bed... login, Gabriel Wicke

Gabriel Wicke wrote: How well does DNS round robin work in practice as compared to a proper load balancer? I've never tried DNS round robin because people say it sucks. But I have no actual knowledge from first hand experience. I do know that load balancing using the tools that linuxvirtualserver.org talks about works great, and gives a very good and predictable level of control. But yeah, there'd be no problem in waiting on that and just using squid, as per the rest of your diagram. --Jimbo

Gabriel Wicke wrote: Here's another way of doing it without the load balancers: have a pool of say 36 IP addresses (chosen because lots of numbers exactly or approximately divide into it: other numbers would do fine), and do simple DNS round-robin to balance user traffic across these. Browsers and DNS caches will cache DNS lookups, but we don't care, so long as traffic is distributed roughly evenly across the IP addresses. Now, have N << 36 Squid servers, and allocate the IP addresses _dynamically_ to the servers. If they are all on the same Ethernet, this can be done using ARP. Very roughly: First set every machine to serve requests on all IP addresses on each Ethernet card in the shared pool. Requests will be seen so long as that logical interface is configured "up" on that machine. Now start a polling process: * every machine checks whether there is a live machine at each IP address in the pool at some reasonable pseudo-randomized interval, by doing an ARP request * if it sees that another machine as well as itself is serving an IP address from the pool, it will take down that logical interface, to prevent allocation collisions * when it sees there isn't, it looks at the number of addresses assigned to each machine (which it knows from its ARP scans) * am I the machine with the lowest MAC address from the set of least-assigned machines? --> take over the IP by calling 'ifconfig' for that interface The interval is pseudo-random to break symmetry to prevent synchronization by entrainment and thus make races very unlikely. (Note that races are self-fixing, see below). Also, excessively heavily loaded machines could also _take down_ logical interfaces, to allow other machines to take up the load. This would also be done on the basis of exceeding a "fair share": a damping mechanism would be needed to stop excess re-balancing and to aid convergence. (Margin too small to contain details of formula for stable damping thresholds and timeouts). Upside: simple and distributed. Race conditions will be very rare (p<0.001), and rapidly corrected. If we poll each IP address once a second from each machine, a new server will be allocated to a fallen IP address within substantially less than a minute (it gets less as N gets bigger). Overhead on each machine: user-space doing an ARP probe once a second, the kernel listening to N ARP requests per second. Downside: when an IP address is transferred from one machine to another, any HTTP transfer in progress will time out. However, as this should only occur when nodes fail, or when new or rebooted nodes are brought up within the cluster, this is probably acceptable. Doing things this way could possibly eliminate the need for the load-balancing boxes completely, and the scripts are no more complex than the heartbeat script. Possible problem: partial failure of a system, so it responds to network requests but not application-level requests Two possible resolutions: * watchdog-like process which performs local tests, and in the event of failure, shuts down all logical interfaces in the pool on that machine, allowing them to be picked up by other machines * remote monitoring, and remote operator intervention via SSH or the power strip, if the above fails (automated STONITH does not make sense...) The code can written entirely in Python, and can simply use standard UNIX commands to do low-level stuff. -- Neil

On Tue, Jan 06, 2004 at 03:09:01PM -0800, Jimmy Wales wrote: DNS round robin sucks because it (i) doesn't provide a way to check for a servers outage and (ii) is not suited for a farm of differently sized servers. Having a heart beat between the two servers solves issue (i). In case of the outage of one server the other server takes over the IP and both round robin IPs point to the same server. Issue (ii) can be addressed by having similar boxes. Since those have to be purchased, this should be easy to achieve. In the office, we ran a website using DNS RR. It was about 500,000 hits per day and 52% of the request hit the "first" server, 48% the second. We changed to HW load balancers recently since the number of web servers increased. ==Sizing== Regarding sizing of the squid boxes: We set up internal squids for web caching and noticed that squid does not support SMP. One squid process can utilize only one CPU. My proposed sizing: 1 CPU at about 2GHz 2 GB RAM 2*36G 15kRPM SCSI Disks Remote controller (rILO, eRIC) for remote "lights out" management 1 Unit rack mountable redundant power supply Being most familiar with Compaq's Intel servers, I configured the box in their webshop and it was about 4,000$ Regarding CPU size: http://web.archive.org/web/20030605225127/hermes.wwwcache.ja.net/servers/sq… states that a dual Ultra Sparc at 170 MHz was able to process 1.8 million of requests per day. This is roughly the current load of en:. A 2 GHz x86 CPU beats the Sparc by far. If we want to spend more money on the box, add memory. Regards, JeLuF

On Thu, 8 Jan 2004, Jimmy Wales wrote: SCSI performance is better in practice too, especially when you have multiple hard disks. But the work those boxes will have to do (load balancers, squid, web serving, mail server) does not seem HD intensive, if they have enough RAM. SCSI disks are usually more reliable, simply because disk manufacturers test them more and use higher quality components (hence the bigger price). If our boxes are redundant, and each box has a RAID 1 array, I don't think that going with IDE/SATA is a problem. Alfio

On Thu, Jan 08, 2004 at 04:37:58AM -0800, Jimmy Wales wrote: The 2,6 GHz is 1$ cheaper. If you want to buy the same boxes for all purpose (squid+apache), I'd go for the faster CPU. The SM-1151SATA has non-ECC memory. Being used to ECC and even hot spare memory, I'm not comfortable with this. 2*1GB is what I'd use, too. Looking at pliny and larousse, from the numbers I've been told, their disks are highly busy, bi and bo are very high in vmstat. Comparing with the above compaq it looks cheap. It has no ECC, no remote management, no redundant power supply, no 15kRPM disks, though. Remote management cards are really useful coping with crashed servers. It starts with being able to read the kernel panic message, being able to power off and on the server, and advanced ones being even able to insert a "virtual floppy" during booting (Remember the crash we had while trying to remotely update the kernel?). http://www.techland.co.uk/index/eric is a vendor-independent RMB. Compaq's ILO-boards are much more powerfull, though. There probably isn't much difference between a SCSI and a SATA interface for the same disk. The difference is mainly in the disk seek time since SCSI disks are available at 15kRPM, while SATA are mostly 7,2kRPM. Strange that the difference is 600$ for one configuration and 170$ for the other. Thinking of Squid, I'd rather spend the money for 2 Gigs of additional memory than for the faster disks. Memory should dramatically reduce I/O load on proxies. 2 additional GB are 762$. You'd need heartbeat for the load balancers, anyway. Doing it on the squids would just save two boxes. Why three squids? Two would be enough, and even one should be able to handle the load if the other one fails. Should the DB backup machine really have a different sizing from the active machine? From what I've seen brion say geoffrin is rather idle, so perhaps yes. On the other hand side, administration would be easier when having two identical boxes, or at least same CPU family. A small Altus 1000 E dual Opteron (single Opterons are not offered by penguincomputing?) with 2 GB RAM (ECC!), 2*80 disk, no CDROM, is at 2,810 $. We could probably get away with two of these for web serving instead of four, and save 2 units of rackspace. 2 squids, single CPU (Relion 1X, 2GB, 2,66 GHz P4, 2*80 GB) 2x 2,150$ 2 webservers, dual Opteron (Altus 1000E, 2GB, 2*Opteron 240 2*80GB) 2x 2,810$ 1 DB Backup box (same) 1x 2,810$ ========= Total 12,730$ Using 4 webservers at $1951 each would cost 2,184$ more, plus 2RU space consumption. Very good point. Regards, JeLuF PS: Sorry if this became to long.

On Thu, Jan 08, 2004 at 03:31:00PM +0100, Gabriel Wicke wrote: Agreed. Jimbo, how much rackspace can we use? Plus one box or plus one who thinks the same? JeLuF

Jens Frank wrote: Rackspace isn't an issue at all. When I asked the world for $20,000, my intention is that this will see us through a pretty steep growth curve for awhile. I'd like for us to setup a system now that's robust and has a very large amount of reserve capacity, so that as far as we can manage, the site will be highly responsive to users for some time to come *even in a case where we have had bad hardware luck*. If two squids are good enough and we can survive on one, then I want three. If 3 webservers are handling the load just fine, and we can survive on 2, then I want 5. If geoffrin dies, I want to have another machine that can take the load without breaking a sweat. --Jimbo

Jens Frank wrote: I hear what you're saying. With the right scalable architecture, adding capacity in the future should be as easy as adding a new server or two, copying over some software, and editing a few config files. It seems obvious that buying too much hardware too early is not good, because cheaper and better hardware will be available 6 months from now. Even so, I don't want us to be too stingy. What I'm hearing loud and clear is that people want wikipedia to be fast, highly responsive, and reliable. We don't want to be sketchy about reaching those goals, we want to embrace those goals wholeheartedly and make sure we aren't sitting on money in the bank while the site lags. --Jimbo

Gabriel Wicke wrote: Oh, o.k., I'm convinced then. 1 squid with a heartbeat friend ready to takeover (and/or load balanced as well) is good. I like the idea of 6 apaches. I want to think more about fileservers today. I'm assuming that those are primarily for the images? Can/should they be more or less the same configuration as the 6 apaches? I think there's enormous flexibility benefits of having lots of identical hardware, *when it makes sense* to do so. When it doesn't make sense, it doesn't. I.E. if we have 1 box that needs to be 'bigger' in some ways, there's no sense in spending money on 6 other boxes just to make them all identical. But if the requirements are roughly similar for different roles, then we shouldn't try to overspecialize the hardware, I think.

Gabriel Wicke wrote: I agree. For now I will forget about separate fileserver machines. --Jimbo

Jens Frank wrote: I agree with this wholeheartedly. The same money now used on "useless" redundancy or "growth capacity" will buy 50% more in 6 months. I am advising everyone in my environment to take always "one step back" in hardware, this will be three steps less in price. Remember that the top of the line at any moment is out of date in two months time anyway, so you can just as well buy 2 months old technology right now. A similar argument holds for memory capacity. I have often seen that people including myself bought an "extensible" computer, but before you actually want to extend it the parts are no longer available or perform badly in your system. If you're really going to be growing memory to 16GB on a system in the future, buying the whole 16GB at that point may be more reliable and cheaper than going for very special modules right now. ECC is slower and more expensive, but it will prevent an eventual memory error from propagating into disk files thereby potentially corrupting data. This might be advisable to a project like Wikipedia, but only on database and/or file servers. The only thing where I have personally seen that "expensive pays" is in SCSI vs. ATA (no experience with SATA): The SCSI protocol is so much smarter that it can be not 2 but 10 times faster than ATA if addressed in parallel by concurrent processes. An ftp server working with ATA that I have used would grind to a standstill whenever a backup was running.... Rob -- Rob W.W. Hooft || rob(a)hooft.net || http://www.hooft.net/people/rob/

Jimbo, will the new servers be installed directly to the new colo or go to California first? JeLuF

Jens Frank wrote: To be sure that I understand. For essentially the same price, you recommend a 2.6Ghz Celeron over a 2.4Ghz Pentium? That sounds fine to me, I just don't know how the differences between Celeron and Pentium affect performance for webserving. Can you elaborate on this? All my recent server purchases have been from Penguin, and they are always ECC memory, which costs more and performs less, but which is supposed to be more reliable. But is it really? How much more reliable? Or have I misunderstood the difference completely? My new G5 uses non-ECC ram, and it seems to work just fine. I have had plenty of bad sticks of ECC ram in my life. Would you consider the use of non-ECC ram to be a "deal killer" on the SM-1151SATA, or just something that's not quite ideal? *nod* I'll look into these. How does a remote management card compare to serial console redirection? I couldn't find a price for that Eric card, what do remote server management cards cost, generally? Keep in mind that we are going to have easy hands-on access to the servers, and we're going to be in a facility that's staffer 24x7 with free "remote hands" service for things like pushing a reset button and typing a few things on the console, sticking in a floppy, etc. I see a difference of $372 and $772. You're right, though, it is strange. (Oh, I see, you were differencing in the other direction, but either way, there is something odd here. I will study it to make sure I didn't make a mistake.) And if this is non-ECC ram, then perhaps 2 additional GB are less than that from other vendors. I want everyone to write things this long, it's the only way for us to really study this out. :-) I will read every word that everyone says, because I have a very strong responsibility to make wise purchases with other people's money. --Jimbo

Gabriel Wicke wrote: Well, the lowest machines that I can find widely available are 2.4Ghz... the state of the art these days is 3.2Ghz. Upgrading from a 2.4Ghz to 2.6ghz is only $28 per CPU (for the celerons). Do you know of a vendor where I could buy 9 2Ghz machines for the price of 5 2.6Ghz machines? Or was that just a hypothetical example? Someone offered to let me in on their deal with a vendor for last-generation Pentium III boxes, at $800 each. For simple webserving, does it make sense to look at those? Obviously we could get twice as many of those for the money... At some point, of course, the management of a lot of servers becomes a hassle. But I don't suppose there's much difference between managing 6 identical webservrs and 12. --Jimbo

Gabriel Wicke wrote: Dell is a bit expensive compared to smaller outfits that specialize in server hardware. I've used penguincomputing.com a lot in the past, with good results. I have 1 server from racksaver.com, it's a good machine, but on their website the other day, they seem to be specializing more in the cluster/beowulf world than commodity webservers. And finally, just recently the CEO of livejournal.com recommended siliconmechanics to me. --Jimbo

On Thu, Jan 08, 2004 at 02:03:19PM -0800, Jimmy Wales wrote: No, I recommend the 2.6 Pentium, according to their web shop, the 2.6GHz version is cheaper than the 2.4GHz Pentium. Without ECC, you can detect a memory fault. Kernel panic. With ECC, you get a message to the log file and can call Penguin to send a new module before it's completely broken. Except for "dead on arrival" I've never had a failure of an entire ECC stick. But might be that the price difference makes a little quality difference, too. The later. ECC is just like redundant power supply. It provides a way to hit reset and power off/power on. But having remote helping hands, we perhaps won't need this. Compaq charges some 600 US$ Yup. Didn't know this when proposing the remote cards. Those were the prices from SM's web configuration tool. JeLuF

One last thought... Jens Frank wrote: Jens priced this configuration at $4000. I was able to replicate the same performance configuration for $2500 or less. I think that for $2000, or half the price, I could match the performance of the box he describes. However, my proposal does not include either remote controllers nor redundant power supplies. Here is my thinking, please think with me if this is right or not. 1. If my server costs $2000, and Jens costs $4000, then I don't need redundant power supplies since I have redundant *servers*. With only a little bit of work to make sure that dead machines are taken out of service, we have a lot more capacity for the same money. 2. A remote controller would be sweet in our current facility. But we are going to move to a facility with the following features -- (1) 24 hour a day remote "hands on" service (2) geographically close to me and my employees so that we can physically drive over (more on this later today or tomorrow after I tour one more facility). --Jimbo

On Thu, 8 Jan 2004, Hr. Daniel Mikkelsen wrote: Any server class motherboard BIOS will have "BIOS Console Redirection". It is rather stupid and crappy, but it *does* work and doesn't cost 1000$. I'm using this on Tyan S2721-533's right now. (the onboard video is disabled.) (And my S2460 dual athlon "desktop" too.) --Ricky

On Thu, 08 Jan 2004 13:13:58 +0100, Jens Frank wrote: The default configuration for using Linux Director would be no load balancing at all and one machine on standby with Heartbeat. We could do the same with the squids, but it makes more sense to use both machine's ram. If money was short i'd propose to buy 4Gb ram and see how much money is left for a fancy hd. I theory the hd should be used only for booting and logging. It might make sense to go single-processor 64bit for the Squids- not for cpu, but for future memory expansion. I think Raid isn't necessary for the squid if the logging is done to a separate disk. Agree completely. My Celeron 2Ghz does about 700 html requests/second at 54% cpu (siege load tester taking up the rest). That's 60.2 million requests per day. Gabriel Wicke

On Thu, 08 Jan 2004 14:06:57 -0800, Jimmy Wales wrote: Yes, but the two Linux Director wouldn't be load balanced because each of them can deal with the full traffic. Same for the Squids, but it wouldn't hurt to use them both. Yes, i'm not against RAID at all- i just wanted to stretch that 4Gigs of ram might be more important than the most advanced five-dics RAID. If we have enough money- then fine. The Squids will do all the logging- so the disk will be accessed regularly. I'm not shure either. 4Gigs will carry a long way i guess (especially with the 'join' configuration of the Squids)- and accessing a few bigger files from disk won't be a performance problem. By the time a memory upgrade would be beneficial a full machine with that amount of ram (of possibly a different kind then) might cost no more. So x86 with 4Gigs might be fine as well- cpu-wise anyway. The question is if this is really needed that soon- if the load really explodes we could start adding mirrors which would also save some bandwidth. Once the envisoned setup is running the next step is mainly about a DNS thing like SuperSparrows. Squid3 is scheduled for release this spring- with ESI this can lower the load drastically again. __ Gabriel Wicke