[Foundation-l] Re: Information flow

[Domas Mituzas]

Hello,

> You were able to solve the problem by adding hardware. Whether 
> that is a victory or a failure is a matter of opinion.

See, a community knows about new hardware more, because 
they pay for it. They do not pay developers and therefore do not 
know what software optimizations were introduced. And we did 
solve many problems by introducing software optimizations. 
So far, operation of Wikimedia cluster is a pure victory.

> Yes, blindly hacking away, disabling useful functions at random, 
> instead of analyzing where the bottlenecks were.  It was very 
> embarrasing to watch.  The growth numbers in 2002-2003 for 
> susning.nu (which still runs on a single server) show what 
> software optimization can do, should you have chosen that path.

What we have now is content produced by massive 
collaboration with a small export of it via our caches 
to the open world. 

Sir, we have profiling information. We know where our bottlenecks are. 
We deal with those bottlenecks. I know how to do software optimization 
for websites, though. Remove as much magic as possible, use very simplistic 
methods, ...We could serve whole website as HTML files from single 
Pentium4 box at gigabit speeds. We could drop accessibility and use 
lots of AJAX for all content loading and interaction. We could use MediaWiki 
as application server instead of script. There are still lots of challenges ahead.
There are ways to do things better. 

If you have any specific ideas on how to 'optimize' all that - feel free. 
Join into #mediawiki or #wikimedia-tech channels, discuss about profiler 
output with developers, brainstorm on how things can be done better.

I also have cases in my portfolio of 'growth numbers', but none of 
them reached the scale of Wikipedia, neither in load, neither in variety 
of users, neither in size of content. And we did have bigger fulltime 
development/operations teams, budgets as well.

Cheers,
Domas