Order 64-bit capable machines, of course, for their security advantages, and speed advantages when running 64-bit software. Move to 64-bit software as soon as reasonably possible.

Want to voice 100% approval of this. Nobody wants to hear that Postgres compiles natively in 64-bit and makes use of it (I measured 40% increase in row-insert performance on UltraSPARC, among other things), but it bears mentioning. I don't know whether MySQL is n64'able. I can help building a 32/64 or a 64-bit gcc if anyone would like help. Opteron, Opteron, Opteron. I don't think we need 64-bit Xeon's or Itanium/Itanium2's.

vastly more reliable systems deployment. Consider Debian as both an operating system and a deployment system: at the moment, different machines run different operating system flavours, making sysadminning harder.

RedHat, of course, has its own products for these purposes. But I'm pretty OS agnostic.

Databases are a different issue: you can't apply the same commodity thinking: however, try to order DB machines in identical multiples, too,

Anyone know what an 8-way 848 Opteron costs these days?

for the same reasons. Clearly the DB machines will need to be hand-crafted. I don't know much about databases on this sort of scale... but I imagine the Wikipedia developers do.

First, multiplicity. Second, fast disk. Hardware raid controllers and tablespaces which allow you to put your indices on the really fast disk and your "big data" on the slower, cheaper, bigger disk. Is SAN attachment an option or are we sticking with NFS? SAN over 1gbit ether (which is where we're at) is not so bad. I mean, it could be worse.

for your servers, and hence your data and syadmin sanity. An ounce of prevention is again worth a pound of cure.

Amen.

Finaly, you might consider buying a cheap radio clock on a per-site basis, if your colo does not already provide you with a local stratum-1

Am I missing something? Can we not just sync with {tick,tock}.usno.navy.mil? If usno.navy.mil goes down, we have much bigger problems than a toasted wikipedia.

Nobody's mentioned backups. Automated backups are hard to do, and manual backups require somebody to go there and swap tapes. Is a tape changer in the cards for us? We'd need to go with LTO or something.... and that's pretty ugly, money wise. Anyone?

aa

However, consider the global supplier situation for Opterons. Alas, Dell don't do them. Who would be the right supplier for this?

HP makes em and supports them (they even support Windows on them in an experimental beta, but I didn't tell you that). Additionally, Sun makes them, and they sell them VERY cheaply. That might be the best route. You don't gotta run Sun Linux 5.0 (or whatever they're calling it these days). Sun sold us (AOL) a dual opteron 242 for $2500 with 2gb of ram and 72gb of 15krpm disk not terribly long ago. Wikimedia doesn't exactly have AOL's relationship with Sun, but it might be possible to call and "talk them down" a little. And the FOSS argument doesn't apply to hardware, right? (because I know we have some Sun and Solaris haters here)

Oh, and don't forget the Gbytes and Gbytes of RAM!

For database servers? Yeah. 4gb minimum. If MySQL supports being built 64-bit, and is designed to take advantage of it, we can reap SUBSTANTIAL rewards from this. I'd say go with 8gb if we can afford it.

DBA's, speak up...

...

Am I missing something? Can we not just sync with {tick,tock}.usno.navy.mil? If usno.navy.mil goes down, we have much bigger problems than a toasted wikipedia.

Consider connectivity going down at a remote site (takes just one backhoe at the site 1/4 mile down the road where all of your supposedly "diverse" fibers join the same duct, or simply a router going mad, or someone hitting the Big Red Power-Off Button at ********* ***** [substitute your national Achilles Heel IXP]), rather than Global Thermonuclear War. Trust me, a local clock is a good thing. That's why it's such a useful service to have onsite.

I don't buy it. I've been syncing to USNO for years, and it Just Doesn't Happen to them. It doesn't happen to the DoD because DISA doesn't let them. NRO and {{spooky agencies}} all use them.

Now, that's a very good point. However, it might be better to do what Linus does and "let millions of people mirror it everywhere". This might be something to ask organizations like to UK Mirror Service, Internet Archive and Google to do on a formal basis. This way, the backups are

On a formal, hourly basis? Or do we decide we're comfortable with a day's loss of data if a nuke falls on florida?

off-site too. The current archive is 50G. If we take a week to back it up, that's a data rate of 50e9*8/(86400*7) = 662 kbps < 1 Mbps. So Wikipedia could perform complete backups weekly to three different sites at a cost of 3 Mbps sustained. That's cheaper than the cost of the tape media.

Weekly doesn't cut it. For full backups, it does. For incrementals, we need nightly or hourly. Does MySQL support live backups? (I understand on-the-fly transactions)

aa (talking systems shit and not getting flamed, yay)

We've been running our MySQL boxen compiled for x86_64 on 4GB Opterons for some time.

Postgres had some issues back in version 7 on some versions of Solaris.

What software are you aware of that doesn't "support being built 64-bit"?

Afterstep had some issues a while ago:

http://sunportal.sunmanagers.org/pipermail/summaries/2003-March/003425.html

My introduction to Linux started in 1999 with LinuxPPC on my Macintosh G3. Lots of bright linux programmers made all sorts of assumptions about endianness and things like xmms wouldn't compile. Programs like Jack the Ripper had assembler in them that would explode spectacularly. These same people are out there, right now, writing code on their 32-bit athlons or whatever, assuming that the rest of the world uses their flavor of linux on their flavor of hardware.

It's nice to see that MySQL doesn't suffer from these problems, but its previous failings (subselects, pl/sql, still lack of ACID, etc), left me in doubt of its ability to build natively n64.

Just been bitten many times by porting software. As I said, my background is with many Unixes. Porting stuff from Linux to IRIX is probably the worst, especially when the linux guy used gcc, and I wanted to use the SGI compiler. But I digress.

Happy for mysql. Would anyone else like to troll me?

aa

How about current versions of relevant software that we use in production? Since you don't even care enough to check whether MySQL ships compiled for AMD64 before spouting off, I guess I shouldn't be surprised.

Brion, if you don't want me here, please tell me so, and please indicate to jwales and chaper that you felt this pissing contest of yours was more important than my input -- which both requested.

I'll just stick to editing {{mil-cruft}} articles and get banned on irc when something breaks next week.

If that's okay with you, that is.

Alex

...

Brion, if you don't want me here, please tell me so, and please indicate to jwales and chaper that you felt this pissing contest of yours was more important than my input -- which both requested.

Input's fine, but it'd be nice if we can get input without having to put up with being told we're all incompetent and U N S C A L E A B L E by someone who won't take a minute to find out about our current development work before declaring it nonexistent, or check if the software he's bashing is available for 64-bit before bashing it on the assumption that it's too backward to possibly work.

Yeah, I'm outta here.

You guys just keep scaling (no spaces this time) the way you've been doing it. I'm sure the weekly outages will stop and the database will magically become multi-homed, and you'll have fancy datacenters in germany and belgium. I'm sure that my donations will help you maintain the status quo, and that i'll be seen as the asshole who came along and made all these inflammatory statements and left in a huff.

So be it.

You'll see me editing {{mil-cruft}}.

Push hard enough, brion, and there's just a point when somebody who is a /volunteer/ will leave. Maybe I'll form a "conscientious objectors" lobby, or maybe I'll just laugh the next time the wikipedia and its related projects are down and go for a walk in the big blue room.

<sound of door slamming />

Alex

Sj wrote:

Doors keep slamming of late... Alex, you were less abrasive and far more helpful than a lot of people who troll the mailing lists, but it did feel as though you were trolling for a while there.

Don't have such a thin skin! I'm not a dev, but I was excited to see your energetic input. There are many people who are thinking about ways to improve the site's performance -- people who spend hours a day doing exactly that. If you make your suggestions politely, especially if you do so on-wiki where they will outlive temporary local disputes, they will be heard loud and clear.

It seems to me that perhaps we need a "wikidev" equivalent of ESR's essay "How To Ask Questions The Smart Way" on one of the wikis, in an appropriate location. Perhaps it could be called "How To Volunteer The Smart Way". A few of the most important points should be these:

* Rather than telling us your qualifications and how you'll change things around here, tell us your skills and make it clear that these skills are available at the discretion of those who come before. If that isn't acceptable to you, consider that you might not be a good fit for this volunteering effort.

* Be humble. We want helpers, not managers.

* Start by asking questions, rather than issuing recommendations. Many recommendations made by people new to volunteering here have already been considered and discarded for entirely sensible, but not immediately obvious, reasons. Many such recommendations are redundant, because we're already considering how best to implement them, or whether implementing them is feasible at this time. Many such recommendations have already been implemented or are being implemented now, and you just don't know about it.

I'm not sure where exactly such a document would go, or I might start working on it myself, but if it never surfaces anywhere else I'll at least stick it on my Wikipedia page eventually to give people an idea of how to approach me, personally, about offering help.

This document is often listed in the irc topic on #plone

http://plone.org/documentation/how-to/ask-for-help

As it states, it is intended to be a user-friendly alternative to Eric Raymond's How to Ask Questions the Smart Way. It might prove a reasonable starting point. Hope it helps.

/jsb

It seems to me that perhaps we need a "wikidev" equivalent of ESR's essay "How To Ask Questions The Smart Way" on one of the wikis, in an appropriate location. Perhaps it could be called "How To Volunteer The Smart Way". A few of the most important points should be these:

• Rather than telling us your qualifications and how you'll change things

around here, tell us your skills and make it clear that these skills are available at the discretion of those who come before. If that isn't acceptable to you, consider that you might not be a good fit for this volunteering effort.

• Be humble. We want helpers, not managers.

• Start by asking questions, rather than issuing recommendations. Many

recommendations made by people new to volunteering here have already been considered and discarded for entirely sensible, but not immediately obvious, reasons. Many such recommendations are redundant, because we're already considering how best to implement them, or whether implementing them is feasible at this time. Many such recommendations have already been implemented or are being implemented now, and you just don't know about it.

I'm not sure where exactly such a document would go, or I might start working on it myself, but if it never surfaces anywhere else I'll at least stick it on my Wikipedia page eventually to give people an idea of how to approach me, personally, about offering help.

-- Chad [ CCD CopyWrite | http://ccd.apotheon.org ] _______________________________________________ Wikitech-l mailing list Wikitech-l@wikimedia.org http://mail.wikipedia.org/mailman/listinfo/wikitech-l

Alex J. Avriette wrote:

...

How about current versions of relevant software that we use in production? Since you don't even care enough to check whether MySQL ships compiled for AMD64 before spouting off, I guess I shouldn't be surprised.

Brion, if you don't want me here, please tell me so, and please indicate to jwales and chaper that you felt this pissing contest of yours was more important than my input -- which both requested.

I'd be happy for help with discussing and researching options from anyone with some knowledge in the area. You offered help, I accepted the offer as an individual, hopefully with the understanding I wasn't doing so as a representative of anyone else. Among the many things that puzzle me about this entire episode, there are three in particular that stand out:

1. I haven't received a reply to my last email to you directly.

2. Someone else contacted me to let me know that he/she had the distinct impression that you thought the two of us (you and I) were on bad terms personally, now, for some reason. I made a couple of informal inquiries, but haven't gotten any information and decided to just let that go for now as rumor.

3. Your references to me and to Jimbo seem to indicate that you feel like you've been appointed some semi-official task, which strikes me as odd. Your reference to both of us requesting your help ("my input -- which both requested") is a good example of that. In your position, I would have phrased that a little more like this: "my input -- which I offered to both, and both accepted the offer." This may seem like a trivial detail at first glance, but the fact of the matter is that you volunteered and we said to ourselves "Hey, another volunteer should be a good thing!" Nobody solicited you as a volunteer. That doesn't make your help less valuable, where it's offered, but your apparent belief that you have some kind of official sanction and that there's a precedence of importance assigned to your input is, as I indicated, puzzling to me.

I'd still like to have your input, where it's thoughtful and free of abrasive qualities that are likely to put off the devs doing sysadmin tasks.

That's just me, though. I don't speak for anyone else in this.

Neil Harris wrote:

Remember that your tightest constraint is admin skills and time; having a highly standardized system for deployment will save you _vast_ amounts of scarce time, and any deployment automation you can use will make for vastly more reliable systems deployment. Consider Debian as both an operating system and a deployment system: at the moment, different machines run different operating system flavours, making sysadminning harder.

Keep recommending Debian. I won't complain.