On Mon, Dec 04, 2006 at 01:15:21PM +1100, Steve Bennett wrote: And rightfully so. :-) Cheers, -- jra -- Jay R. Ashworth jra(a)baylink.com Designer Baylink RFC 2100 Ashworth & Associates The Things I Think '87 e24 St Petersburg FL USA http://baylink.pitas.com +1 727 647 1274 (I've been informed that the 'divorce' quote makes me look misogynistic, even though Jennifer Crusie is (pretty obviously) a female romance novelist. So I've removed it. :-)

An order has been placed for an additional six database servers. The servers should be delivered Tuesday or Wednesday this week. On 12/3/06, Steve Summit <scs(a)eskimo.com> wrote: -- Brad Patrick General Counsel & Interim Executive Director Wikimedia Foundation, Inc. bradp.wmf(a)gmail.com 727-231-0101

Jeff V. Merkey wrote: I run a Wikipedia instance on the 366MHz OLPC XO-1 (http://dev.laptop.org/~krstic/xo); it works because I'm the only user. Neither storage nor disk throughput help when your problem is sheer load, and you're computationally-bound. -- Ivan Krstić <krstic(a)solarsail.hcs.harvard.edu> | GPG: 0x147C722D

On 12/4/06, Jeff V. Merkey <jmerkey(a)wolfmountaingroup.com> wrote: Are there links for the capacity planning and performance testing that's been done so far on production servers? -- -george william herbert george.herbert(a)gmail.com

Jeff V. Merkey wrote: These numbers do not take into account the constant DOS attacks and hacker attempts I get from mainland China. I have the servers setup on dynamic IP ranges which switch between several IP subnets every other week to allow folksin China to get access. Most of the traffic, unfortunately, are attacks designed to kill my servers from China. They have been having a hard time of late since I setup the SSH ports to jump around between ranges other than port 22 after 3 or more failed attempts to login via SSH. Jeff

On 12/5/06, Jens Frank <jf(a)mormo.org> wrote: Is that 22,000 requests per second page requests, or total hits per second? Is there an edits-per-second statistic available? I went wandering but didn't find one... -- -george william herbert george.herbert(a)gmail.com

On 04/12/06, Jay R. Ashworth <jra(a)baylink.com> wrote: Let's filch a couple supercomputers. Rob Church

On 12/4/06, Jeff V. Merkey <jmerkey(a)wolfmountaingroup.com> wrote: Why filch? Call up and order... -- -george william herbert george.herbert(a)gmail.com

On 12/4/06, Rob Church <robchur(a)gmail.com> wrote: I would expect that a little effort would result in Wikipedia finagleing something like the educational institutiton discounts out of most vendors. Do you have any idea how deep those discounts typically are? -- -george william herbert george.herbert(a)gmail.com

On 12/4/06, Alphax (Wikipedia email) &lt;alphasigmax(a)gmail.com&gt; wrote: <brief response> "No" </brief response> <computer architect hat> The problem with distributed computing for general computational problems requiring interactive updates between computational nodes is that the amount of traffic between nodes increases both with the amount of updates and with the number of nodes. The various @home type projects, and batch job type grid computing, are efficient network-wise; there is little to no network interaction other than submission and results. Where data has to flow laterally, as in a database system, the scaling increases as described above, rapidly bringing the distributed application to its knees. This is a known problem in the design of MPI type supercomputer clusters. As the required interconnect communications bandwidth increases, the optimial solution moves from highly distributed, towards larger nodes in a higher speed network, towards a few nodes in a very very high speed network, and finally towards a single large SMP computer. If you are pushing the interconnect very hard, you're having to pay for extremely high performance interconnects, and ultimately there's no reason not to just buy large SMP systems with single system image and more compact footprints. The highest speed system to system interconnects cost as much as the SMP system components do. One current commercial example of this particular problem is the throttled Oracle RAC problem, where you spread a big database out over a large pile of Linux nodes, but the database update rates overwhelm the private network interconnect for the shared cache updates. There are plenty of HPC computer horror stories of equivalent problems out there in the field for other workloads. There is no single right answer; even the large labs who have large quantities of professional computer architects and code wranglers working on optimized highly multi-processor projects don't agree on how to do most work, which is why there are still a bunch of competing supercomputer/supercluster vendors out there, in many cases individual centers or labs buying from multiple vendors as they have different types of work in progress. For Wikipedia's database problems, I would have to have about 10x more info than I currently do to make particularly useful actual advice or suggestion, but I can guarantee you that WikiDB@home would be a disaster of heroic porportions 8-) </computer architect hat> -- -george william herbert george.herbert(a)gmail.com

On 12/4/06, Ivan Krstić &lt;krstic(a)solarsail.hcs.harvard.edu&gt; wrote: You do realize that Google has spent the better part of a half billion dollars on engineering a completely ground-up distributed system software architecture, working with a problem that (unusually among largescale enterprise data management) can theoretically be efficiently partitioned? The reason not everyone is using Google-like IT architectures is A) That they can't afford to develop it, and B) That in most cases, that architecture would be worse than what they have now, given that most IT problems don't partition so well... -- -george william herbert george.herbert(a)gmail.com

On Tue, Dec 05, 2006 at 12:52:50AM -0500, Ivan Krsti?? wrote: For what it's worth, *my* perception of what jeff said parsed as "buy *six* powerful appliances". :-) Or, more generally, COTS is wonderful, but is it cost effective now to spec one or two levels higher of COTS than we have been? Cheers, -- jra -- Jay R. Ashworth jra(a)baylink.com Designer Baylink RFC 2100 Ashworth & Associates The Things I Think '87 e24 St Petersburg FL USA http://baylink.pitas.com +1 727 647 1274 (I've been informed that the 'divorce' quote makes me look misogynistic, even though Jennifer Crusie is (pretty obviously) a female romance novelist. So I've removed it. :-)

George Herbert wrote: Wikipedia can partition the database across languages (as it does already with the largest ones), and when individual languages grow to be too large for a single server to deal with, there are other partitioning schemes to look at. So it's a bit simpler here, as it's not one monolithic data store that's growing without bound. And yet an even larger social networking site continues to happily churn along with MySQL. Clearly there are examples each way, but in the case of WMF, there are also principles that factor into the equation. -- Ivan Krstić &lt;krstic(a)solarsail.hcs.harvard.edu&gt; | GPG: 0x147C722D

Hi! Indeed, selling big iron is more profitable for vendors than those commodity pizza boxen. It all depends on data layout, structure, application design. Any application out of the box may have scaling issue, but sometimes it is possible to do some horizontal magic and you're given lots of fresh air to breathe. Oddly enough, I've heard quite a lot of stories where maintaining oracle environment was too expensive and switching to mysql was the solution. There's nothing really interesting - lots of requests are being served even in front of application layer. Out of what is left is being handled by lots of apaches and quite distributed data system. We don't do much edits, so there're no more than few hundred update queries per second (really active updating happens in lossy databases, e.g. memcached). Replication works, and we send MySQL read queries to slaves. We do up to 40000 db requests per second. We do not have enough of funding to have thousands of linux boxen, so we end up doing what we can with current resources. *shrug*, they had performance issues, but IIRC they solved them. -- Domas Mituzas -- http://dammit.lt/ -- [[user:midom]]

On 12/5/06, Ivan Krstić &lt;krstic(a)solarsail.hcs.harvard.edu&gt; wrote: I think you meant "that WikiMedia doesn't". We don't have access to Google's technology. We don't have [[MapReduce]]. Steve

On 12/5/06, Rob Church &lt;robchur(a)gmail.com&gt; wrote: If there's anything currently in Google which would seriously benefit WMF, other than cash, it is likely that we could approach the right people and get access to it. I know some midlevel people to start asking questions of; Jimmy and others at the board level probably have more appropriate high level contacts. MapReduce doesn't seem relevant to the largescale MediaWIki deployment problem. Is there any Google tech which anyone specifically thinks would help? Or generalizing, is there any commercial or semi-commercial tech in the world which would seriously help? -- -george william herbert george.herbert(a)gmail.com

On 12/5/06, Ivan Krstić &lt;krstic(a)solarsail.hcs.harvard.edu&gt; wrote: Google has emitted quite a bit of IP as freeware; it depends on whether it's a key commercial competitive advantage for them or not. I hadn't been familiar with BigTable, which is sort of annoying since I'm rather familiar with [[Sybase IQ]], the commercial column-based database, and I've done a bit of evangelizing of column-based databases since I heard of the idea. I do know of GFS rather well, from the technical papers level on up. On first glance... GFS doesn't seem relevant to Wikimedia Foundation work. GFS is all about giving common access to a very large, petabytes scale disk store across a wide sea of systems. As I understand it, one whole database+static content en.wikipedia dump is less than a terabyte, and can fit on the local disks of a single 1-2U rack server. There's no reason for there to be a giant shared filesystem if the dataset fits on one system's local disk. Am I missing something? A column based database (generalizing here) seems like a possible match for our needs; in general, as I understand it, the wikipedia servers are getting hit 99% plus reads, many fewer edits, in terms of the database access? If that's true, then the database contents are more like a data warehousing job (few updates, predominantly read operations), and column based databases seem to be around 10x faster for data warehousing work. The question I would have is whether the database scale is large enough to justify that. If you can keep the indexes for the key tables in RAM all the time, and given the only-few-hundred-GB database sizes now, I would hope that we could, then the index in RAM beats disk access to a column in terms of performance, and the database's disk layout is therefore a second order effect. Can WMF admins confirm that the DB servers are effectively keeping the DB indexes cached in RAM now? -- -george william herbert george.herbert(a)gmail.com

On 12/5/06, Ivan Krstić &lt;krstic(a)solarsail.hcs.harvard.edu&gt; wrote: Well, obviously BigTable depends on GFS, now that I'm looking at it, but it's not a "we could use GFS for GFS'es sake" situation, unless I am missing something. And as I mentioned, BigTable seems not useful as long as our indexes fit in RAM, which I hope is the case. -- -george william herbert george.herbert(a)gmail.com

Hi! If BigTable (whatever it is) would be some magic silver bullet, Google would not use InnoDB or MySQL or whatever. We have few hundred gigabytes of RAM available in cluster. I hope we fit. :) -- Domas Mituzas -- http://dammit.lt/ -- [[user:midom]]

Hi!!!!! Well, this is why we end up with different slave copies of different masters. Oh, and wait, we even end up with different load patterns. Different storage patterns. And we still don't buy 32GB machines. \o/ -- Domas Mituzas -- http://dammit.lt/ -- [[user:midom]]

On 05/12/06, George Herbert &lt;george.herbert(a)gmail.com&gt; wrote: This is where I confess to embedding some sanity in my otherwise ignorable message... Rob Church

Having a link to fundraising in this error message wouldn't be such a bad idea... Bence Damokos On 12/4/06, Rob Church &lt;robchur(a)gmail.com&gt; wrote:

On 12/5/06, Rob Church &lt;robchur(a)gmail.com&gt; wrote: Heh. I really ought to make a photo gallery of all the different errors I've seen at Wikipedia. I must have seen half a dozen really different error screens by now. For my own curiosity, what does "can't" mean in this context - is it a "can't spare the resources to do so", or "doesn't have enough information", or "happens at too low a level"? Steve

On 12/5/06, Simetrical &lt;Simetrical+wikitech(a)gmail.com&gt; wrote: Ah, that makes sense, thanks. Steve