Hello,

Hi. Fabulous colleague-shareholder-report you have there; but could you fix the transition speed on your powerpoint slides? and the font in the footer...

Transition speed? Powerpoint slides? That is a workbook, not a presentation :) people read it as a pdf book! :)

Is this a general trend -- remote hands not being available during critical moments -- or a chicken-and-egg issue with other elements (such as Rob being around)?

It gets complicated with all operations folks being in Europe - hardware provisioning, reinstalls, etc - is usually managed not from U.S. That limits the time window for various work. Datacenter ops are more fixed in time than anything else - and when we needed things to do ASAP, people were in schools/jobs/etc.

Now with Rob around we can be pretty sure that any critical issue can be dealt with swiftly, and non critical jobs still in reasonable time.

This is cool; I had no idea. Is there a longer description of how it works?

Probably Mark could tell much more, but generally equal providers like to exchange their traffic for free - they hop on to traffic exchanges (think of a huge switch, well, in real world it is bunch of big switches :), and look for peering partners. Usually it is a bit of trouble for content providers to get peering, but Google has it with nearly every major provider. For smaller people like us we have to be really cool. Mark did lots of social work to present us as cool in networking world, and we are allowed to use some free resources.

The press release about such activities in Amsterdam was at: http://wikimediafoundation.org/wiki/AMS-IX

Any offers of support from multiple really big donated hosting places?

The big problem is that it has to have hardware - lots of it - to support our cached data set, and if you try to disperse it over multiple datacenters, really complex problems start like how to balance the requests so they go to datacenter which would have that data. Stuff like 'lets have french go here, germans there' adds lots of administrative work - from maintaining all the platform, to actually troubleshooting.

Generally, more datacenters are there, more probably we'll miss problems. This was especially seen by moving Asian languages to Asia - by having platforms we manage less than Tampa we'd eventually end up with them working slower, more errors, etc. What was interesting - no Asians would come and tell that to us - it seems that everyone is used there to bad international sites. Once we reduced the complexity and did just what was nice, easy to manage and efficient - we ended up having people in those countries tell 'yay it is very fast, faster than local sites'.

So, if someone would come up with big donated place, that would bring as much caching hardware as we have now in Amsterdam, or in Tampa - probably it could be possible to consider. Still, for many of these places to host us would be as expensive as cover our costs. And by using more sites, our costs increase.

Are there recent stats on the # of reusers, sites, contributors; mediawiki extension variants / repositories outside the main tree?

Well, quite a few people commit extensions to our repositories: http://svn.wikimedia.org/viewvc/mediawiki/trunk/extensions/

The community outside our #mediawiki cabal has not formed, though there're various extensions floating around, that are more suitable for other sites, than us. And there're lots of small wikipedias or wikimedias all around the world (I run one at job, management never called it 'wiki' - it sounded as old not very nice software, or mediawiki).

Ditto for stats on # and quality of patches from colleagues from other shops. Is there a wall of huge-site-heroes for those who release their patches?

I mean more the knowledge sharing with people who run big non- mediawiki sites. Like, folks from Yahoo helped with APC. Google have nice patches for MySQL. SixApart help with memcached. Even though they don't directly fix our bugs, their engineers are willing to communicate, discuss the operations, and sometimes improve software we use. And in the end they even can buy you a drink. :)

Best regards,