On Tue, 21 Dec 2004, Jimmy (Jimbo) Wales wrote:
Ricky Beam wrote:
Comparing MAC addresses everytime you need to
know where something
is attached is very time consuming and error prone. *Maintain* the
documentation. That's pretty easy as there's only one monkey movin'
cables.
Well, one problem we have is that the monkey (me) gets called out of
the country with some increasing regularity. Another problem we have
is that another monkey (Aaron, at the colo) is going to untangle the
current mess of wires and neatly tie everything into the rack
properly, but is waiting at the moment for us to decide about a switch
solution.
I certainly agree that maintaining documentation is critical here, but
at the same time I think it's pretty important that we are able to do
this maintenance *remotely*. Certainly, I don't think anyone is
advising that we "compare MAC addresses everytime we need to know" --
rather it's just that for debugging/troubleshooting/oddities, it will
be great for us to have the *ability* to figure out which mac address
is plugged in where.
It's not my intent to shame Jimbo (and any other wire pluggin' monkeys.)
What we're dancing around here is called "Change Management" in the
Enteprise. Having been away from the "Enterprise" for a little over a
year now, I've blocked out the nightmare of proceedures and paperwork,
but they exist for a very real reason... at some point, all the cooks
in the kitchen need to know where everything is. Wiki passed that point
some time ago.
It's very simple and easy to record what's plugged in where at the time
it's plugged in. After the fact, chasing cables, is No Fun (tm). I've
worked in the Telco/ISP world long enough to have a degree in Not Fun --
even in regulated spaces, the cabling standards are not always followed
(the tracer line doesn't help if it isn't plugged in, or no one ran power
to the DSX panel, or some [censored] monkey steels the fuse for that panel.
Those two who've replied w.r.t. their "exception" networks don't have
the
breadth of experience that they may think they do.)
As for virtual network redesign... I would like to go on record saying that
it's a Bad Idea (tm), as it's very easy to break the network in ways one
cannot undo remotely. (see also: "haphazard network growth") It takes
a great deal of care, patients, and planning to execute without incident.
Sitting here in my apartment ("warehouse"), I'm several hundred miles from
the colo (and not on the approved list.) If I were to break something, it
could take hours to get someone there to fix it. (It'd take about 13hrs for
me to drive to the colo :-)) And let's face it, remotely, there's never
just *one* person doing something.
Granted, I've done this sort of thing. However, I've always done so with
extensive planning and people on-site available to undo what I'm about to
do -- i.e. power cycle the hardware. And yes, even with planning, I've broken
the internet more than once -- 'tho never for very long, unlike some of
my former coworkers who once left the ISP's network split in half (read:
"fux0r3d up") for ~4hrs waiting for the techs to show up @ 7am and reset the
router... they didn't call the on-call tech, they didn't dial into the POP,
nor did they bother to login to our side of any of the uplinks (ssh access
was not filtered for just such emergencies.)
We're looking at a cost differential for 48 ports
of roughtly $1500
versus $3500. Even if we grow to 10x our current needs (480 ports
added, let's say) the total cost differential would be "only" $20,000.
Wiki will need real Enterprise hardware long before that. In my opinion,
at around 100 ports, it's time to stop hanging things off 79$ switches.
The more switches in the mess, the greater the odds one of them will fail.
I've seen about one 3com 10/100 hub fail every 2 months in the Enterprise
Desktop world -- 20-30 per closet across 9 floors of the office building.
(And I've seen one Acton hub *physcially* catch fire. One of the Charlotte
switch tech drove it back to Raleigh for me. It sat like a trophey outside
my desk for months. One of the PC Techs took it when I left.)
I don't think Wiki wants to look like the mom-n-pop ISP in Mooresville, NC,
I was told about by a USR Sales Engineer many years ago... Imagine, if you
will, a garage with over 100 USR Sportster desktop modems layered on 4x8 ft
window screens stacked 7-8 layers high with a 4ft diameter, roof mount attic
fan on top of them. I wish Jim had taken a picture of that rig :-) It beets
Interpath's (ok, it was called "Global Radio" at the time) "Rabbit
Cage" circa
1993 -- sheet metal frame to hold the circuit boards from 8 Microcom ES
desktop modems running to a Mac II(?). When I started there in late '95,
Charley showed me the Rabbit Cage -- "where we started". That had me
laughing all week. (Yeah, I showed USR the cage in late '97; he was equally
ammused. I think Charley still has that thing.)
Yes, I agree with this -- near-constant, rapid,
semi-haphazard
planning. But at this point, it isn't clear that we have a *lot* of
choice about that.
Nope, that soup is already on the floor. And given the rate of growth,
planning an expandable network to last more than 6months will be difficult.
And wiki will out grow the current mess about twice before the planning would
be done. I'm talking about several months of planning with several more of
begging vendors for hardware. Wiki will be 501(c)3 by then.
Additionally, it occurs to me that this current
discussion should be
viewed - in part - as an attempt to avoid haphazardness. The easy way
forward is the blind way forward: "Oh well, out of ports, buy another
couple of cheap switches". What I'm hearing from people, though, is
that buying more capable switches will make it easier to do things
rationally going forward.
At this point, a few more cheap switches is the best-cost option. When one
needs ports *now*, it's too late to invest time in proper planning. Remember
the 6P thing... Prior Planning Prevents Piss Poor Performance. (I've not
heard that since high-school :-))
The need/requirement for gigE ports to most/all machines complicates matters
by greatly increasing costs and limiting the market. I've built medium sized
(~100 port) networks for about the cost of one 2948G. (also built an ISP POP
for under 100k$, got a chunk of marble for saving the company a bank worth of
money.) But that's a 10/100 network with only 3 gigE ports.
(And yes, I say this knowing full well that there's
an alternative,
which is for Monkey Jimbo to go over there and document everything and
do all the rewiring himself and instruct the colo never to touch
anything. But realistically having an option that doesn't involve me
personally as a bottleneck is always a good idea...)
Just document the wiring when the wire is installed. How many people sit on
IRC while your at the colo? It's not hard to tell "us" what you've
plugged
in where, from which one of "us" can update the docs (both those in wiki and
on the machines.) Unfortunately, this is part of the soup that's on the
floor. It wasn't recorded when it was plugged in, so now, someone will have
to physically trace the cables. (Have I mentioned how much I hate doing that?)
There's no escaping it, it would appear.
--Ricky