Gerard Meijssen wrote:
Kate and all others, Considering that everything is done on such a LOW budget it is sheer magic. I am one of those people that "lurks in the background" offering little or no advice. What people do not realise is that in a "professional" setting solutions are often bought. You buy the latest doda, you hire that magic expert who comes to do his trick again. With the current budget you have a lot on your plate not only PHP but also Apache, not only Apache but also Squid, not only Squid but also MySQL, not only MySQL but also ..
I said on the village pump that I thought we are doing a great job, but I don't think any of the developers I was aiming that comment at read it. Let me take this opportunity to thank everyone and recognise what we have acheived.
We've been working in adverse circumstances. During peak times, the site has been extremely heavily loaded and unstable. Any slight error in misconfiguration, or inaction at a particular time when action should have been taken, causes the site to crash. I've lost count of how many problems we've identified and fixed just over the last few weeks.
Many users say "the developers are doing a great job", or "we all know that the developers are very busy", but the fact is that 99% of users don't have a clue what we are doing. They don't know what our achievements have been and they don't know the challenges we face. Gerard's comments are certainly refreshing in this regard, but I think I can add to them. Assume I am speaking on behalf of the users, since I'm sure every user would agree with me if they only knew who to thank.
Big thanks go to JamesDay, who almost single-handedly administers 8 database servers, a task requiring constant monitoring and work on the order of hours per day. James's advice to the MediaWiki developers and other system administrators is invaluable.
Also on the topic of database administration, Kate's servmon and WikiServices bots which have kept the site running when otherwise it would have been choked with long-running queries.
Thanks to Med and Submarine for their work in network and hardware administration of the Paris squids. Well done Mark and Kate for getting PowerDNS up and running and thus getting the Paris squids into service.
Domas's setproctitle() patch is amazing and we all know it. Of course his other system administration and development work is greatly appreciated.
Thanks to Brion, JeLuF and Hashar for their tireless and usually unrecognised work in fixing MediaWiki bugs.
Thanks to Kate for setting up Pen and Perlbal. This is the third time I'm thanking Kate and that's no coincidence - if she left us we'd be left with a dozen pieces of software that no-one else knows how to use.
I know you're all stressed, all we seem to get is complaints despite what we've acheived. I decided after Caroline Ewen's post on wikipedia-l that I can't afford to answer every single question asking "why is the site slow" or every report of "I'm getting backtrace errors!" The fact is that the site has grown to such a size that every time something goes wrong, we can expect a flood of complaints and queries. My advice would be to answer only some of them, and let alert users distribute the information to everyone who asks. Or just ignore them -- remember your time is valuable. Think of what you could have achieved in the time it took you to put a single user out of their ignorance.
When the public forums are too noisy with uninformed speculation, let's organise what we need to make the site better in less visible, more constructive forums, and work with the Board to make it happen.
-- Tim Starling