Jim Guide wrote:
I was a developer a while back for the (purportedly) 17th busiest website in the world and though the sysadmins were more directly involved in improving response speed, I ended up doing a lot of stuff myself.
Eh gads! We really need somebody with your experience. Please sign up for Wikitech-l at http://mail.wikipedia.org/mailman/listinfo/wikitech-l
The way that Wikipedia is slow is very similar to problems we had. The delays are almost all in the initial request for new pages.
I am not a developer, but I've gotten the same sense by reading/skimming Wikitech-l (checking to see if links are active or edit also was a big drain until we started to cache pages for non-logged-in users).
Once the connection is made, content usually comes across rapidly.
Nod. At least for non-logged-in users accessing cached pages...
This usually points to some sort of full queue in software, or a full queue due to excessive connections on a single machine causing a hardware wait state.
We often get "too many connections" errors and Apache has to be restarted often. I also remember the developers having to increase the number of connections allowed setting a few different times.
[lot of technical stuff and seemingly good suggestions]...
I'm not sure what your current server setup is,
Two machines: ref http://mail.wikipedia.org/pipermail/wikitech-l/2003-September/005976.html
Pliny (supposed to be just a database server)
Software: MySQL, Apache 1.3.27, Linux kernel 2.4.20 Tasks: Host all wiki databases, crunch database queries, webserving the non-English Wikipedias, Wiktionary, meta, Wikiquote and Wikibooks. Hardware: Tyan S2462 motherboard 2 Gig RAM (4 512Meg modules of ECC Reg. 266 MHz PC2100 DDR RAM in 4 slots) 2 Processor AMD Athlon MP 1800+/266FSB Adaptec AIC-7899P SCSI U160 controller 1 IBM 36 GB SCSI Ultra 160 / 10 K RPM 1 IBM 36 GB SCSI Ultra 320 / 10 K RPM (running as U160, of course) Dual onboard 3Com 10/100 (3C982) adapters
Larousse (supposed to the webserver for all wikis)
Software: Apache 1.3.27, Linux kernel 2.4.20 Task: Webserving the English Wikipedia Hardware: TYAN S2518UGN motherboard 2Gig RAM (4 512 Meg modules of ECC Reg. 133 MHz PC133 SDRAM in 4 slots) Pentium III 866MHz processor w/256K cache Adaptec AIC-7899 SCSI U160 controller 1 Quantum Atlas 18 GB SCSI U160 Drive, currently unused and known to be troublesome (awaiting some testing) IBM 36 GB SCSI U160 Drive Dual onboard Intel 10/100 (eepro) adapters
http://mail.wikipedia.org/pipermail/wikitech-l/2003-September/005976.html also has info on our short term and intermediate term upgrade plans. Oh and you may find this bit from Jimbo interesting;
"..[short term upgrades will leave] us with two nearly identical machines, and thus well-poised for these two to be equal load-balancing frontend webservers in the future, when we may buy a MEGA MONDO db machine."
So I think we are already onto your line of reasoning. IIRC we have the money for the short term upgrades, and hopefully will have the US$6,000 needed for the 'MEGA MONDO db machine' after a few more months of donations.
[more great stuff for wikitech-l]...
Cheers!
-- Daniel Mayer (aka mav)