Brion Vibber wrote: Apparently an 800-amp breaker in the main PDU failed. The database servers are currently starting up; this takes a little while, but hopefully should be all done within the hour.

I thought each db server was on a UPS? If not, then why not? I also thought we had the capability to go read only from the other server farms when the Tampa farm went down. If not, then why not? We are a top 20 website that makes more than enough money to hire enough people to make sure services decay gracefully and to make sure we have adequate redundancy. I'm working on the current year budget. Please start thinking about your staffing and equipment needs for the rest of the year.

Apparently an 800-amp breaker in the main PDU failed. The database servers are currently starting up; this takes a little while, but hopefully should be all done within the hour.

Hi! Now as we have enough of money to perform better, we can start planning. First of all, for 24/7 operation we'd need to have 9 fully qualified system administrators ( for every 8h timeslot with redundancy, leave/ sick coverage, etc) (payroll costs ~100,000$ each). Of course as these guys would be working on site operations, they'd not have any time for development. So we need >100 guys outsourced to India (payroll costs ~1000$ each). Sure, as open source technologies are not that suitable for round the clock operation, we might have to start using either Java (enterprise!!!) or .Net/C# (on Microsoft platform, sure, enterprise again!!!). Then, datacenters. To have a reasonably working read-only stand-by datacenter, we'd have to invest ~250k$ in it (with current load). We should of course have 3rd party consultants to help this set up (+100,000$ one time, one week gig). Moreover, we should invest into security measures (as wikis are often hacked). We'd need to screen traffic, ensure highly secure AAA subsystems, uh oh. I guess other developers (especially with experience in high- availability environments) could add more requirements for fluent operation! So... let's start rolling!!! Domas _______________________________________________ Wikitech-l mailing list Wikitech-l(a)wikimedia.org http://mail.wikipedia.org/mailman/listinfo/wikitech-l

On Mon, Apr 10, 2006 at 10:27:23AM +0300, Domas Mituzas wrote: Hee. On a serious note, though, if a backup site is beeing looked at, we might want to talk to the DirectNIC people in New Orleans...