[Wikipedia-l] Why Wikipedia Runs Slow - A Suggestion

[Daniel Mayer]

Jim Guide wrote:
>I was a developer a while back for the (purportedly) 
>17th busiest website in the world and though the 
>sysadmins were more directly involved in improving
>response speed, I ended up doing a lot of stuff myself. 

Eh gads! We really need somebody with your experience. Please sign up for 
Wikitech-l at http://mail.wikipedia.org/mailman/listinfo/wikitech-l

>The way that Wikipedia is slow is very similar to problems 
>we had. The delays are almost all in the initial request for 
>new pages. 

I am not a developer, but I've gotten the same sense by reading/skimming 
Wikitech-l (checking to see if links are active or edit also was a big drain 
until we started to cache pages for non-logged-in users). 

>Once the connection is made, content usually comes 
>across rapidly. 

Nod. At least for non-logged-in users accessing cached pages... 

>This usually points to some sort of full queue in software, 
>or a full queue due to excessive connections on a single 
>machine causing a hardware wait state. 

We often get "too many connections" errors and Apache has to be restarted 
often. I also remember the developers having to increase the number of 
connections allowed setting a few different times.  

>[lot of technical stuff and seemingly good suggestions]...

>I'm not sure what your current server setup is,

Two machines: ref 
http://mail.wikipedia.org/pipermail/wikitech-l/2003-September/005976.html

Pliny (supposed to be just a database server) 

Software: MySQL, Apache 1.3.27, Linux kernel 2.4.20 
Tasks:     Host all wiki databases, crunch database queries, webserving the 
non-English Wikipedias, Wiktionary, meta, Wikiquote and Wikibooks.
Hardware:
Tyan S2462 motherboard 
2 Gig RAM (4 512Meg modules of ECC Reg. 266 MHz PC2100 DDR RAM in 4
slots) 
2 Processor AMD Athlon MP 1800+/266FSB 
Adaptec AIC-7899P SCSI U160 controller 
1 IBM 36 GB SCSI Ultra 160 / 10 K RPM 
1 IBM 36 GB SCSI Ultra 320 / 10 K RPM (running as U160, of course) 
Dual onboard 3Com 10/100 (3C982) adapters 

Larousse (supposed to the webserver for all wikis)

Software: Apache 1.3.27, Linux kernel 2.4.20
Task:       Webserving the English Wikipedia
Hardware:
TYAN S2518UGN motherboard
2Gig RAM (4 512 Meg modules of ECC Reg. 133 MHz PC133 SDRAM in 4 slots) 
Pentium III 866MHz processor w/256K cache 
Adaptec AIC-7899 SCSI U160 controller 
1 Quantum Atlas 18 GB SCSI U160 Drive, currently unused and known to be 
troublesome (awaiting some testing) 
IBM 36 GB SCSI U160 Drive 
Dual onboard Intel 10/100 (eepro) adapters 

http://mail.wikipedia.org/pipermail/wikitech-l/2003-September/005976.html also 
has info on our short term and intermediate term upgrade plans. Oh and you 
may find this bit from Jimbo interesting; 

"..[short term upgrades will leave] us with two nearly identical machines, and
thus well-poised for these two to be equal load-balancing frontend
webservers in the future, when we may buy a MEGA MONDO db machine." 

So I think we are already onto your line of reasoning. IIRC we have the money 
for the short term upgrades, and hopefully will have the US$6,000 needed for 
the 'MEGA MONDO db machine' after a few more months of donations. 

>[more great stuff for wikitech-l]...

Cheers! 

-- Daniel Mayer (aka mav)