> "Awesome. Even given aggregate datasets from jobs, I think we're probably okay for space for at least 6 months. That's plenty of time evidence the value of analytics, and make a strong case for more disks. July is ~8 months away, so the timing lines up perfectly."
Forgive me for seeing this as somewhat less than awesome.
Did we scale down on initial investment? In the hardware planning of March the system was envisioned to "still handle 2013 traffic growth".
If our fundraiser exceeds expectations as it usually does, you're probably right, extra expenses will be approved. However that is still unknown.
If we would sample say 1:10 (and have 1:1 for limited ranges) we would have enough storage for years. Just reminding.
I'm not qualified to set overall priorities, but this is the time to vet basic assumptions.
Or at least make budgetary consequences a bit more explicit.
Erik
From: analytics-bounces@lists.wikimedia.org [mailto:analytics-bounces@lists.wikimedia.org] On Behalf Of David Schoonover
Sent: Tuesday, November 13, 2012 8:07 PM
To: Analytics List
Subject: Re: [Analytics] Kraken Hardware Assignments
Awesome job! A couple of comments on the allocations:
- NameNode
1. I think we definitely want to run the NameNode on a Cisco machine -- NN latency effects responsiveness across all of HDFS as it's used to address everything. We're also running the menagerie of Hadoop-related external dependencies on this machine (e.g., a MySQL for Hive and Oozie, etc). I'd prefer not to tempt fate by co-locating anything with the primary NameNode.
2. an01 is currently has our cluster's public IP, and thus hosts a bunch of utilities, making it a DDoS waiting to happen. We should move the NN to an10 (or whatever other Cisco machine pleases you), as that's way easier than moving the IP. Which brings up...
4. NameNode Frailty. Generally speaking, I think "NameNode is Hadoop's SPOF" tends to be a bit overblown. Our cluster isn't on the critical path of any client-facing operation. It's also not terribly big, so any hardware dedicated to a hot spare (via AvatarNode (iirc) Facebook's High Availability NameNode or nfs or whatever) would cut into the jobs we can realistically service on a daily basis. Once we're sure we have the metal to stay ahead of demand we should revisit this, but I don't think we're there yet.
4. The Secondary NN is essentially the spare tire in the trunk -- it [sadly] gets no load so long as the primary is up -- so we can probably run on a R720 alongside the Hadoop Miscellany and the ETL Nimbus (Storm's JobTracker).
- Kafka. I think we should probably move our primary Kafka brokers to Cisco machines for the same reason I agree we should run ETL on those machines. Both systems are hard-realtime--they need to run ahead of the incoming data stream or we'll be forced to drop data. The more RAM, the more cores, the better; headroom here is important as load will be variable.
- Monitoring. We plan to run both Ganglia and some kind of application-level JMX monitoring. Though these services tend to use a decent chunk of network, they're not otherwise terribly resource hungry. We can probably get away with sticking both on a R720 and otherwise reserve that box for staging and other ops utility work.
- Storage
If we snappy compress and do more rounding up, that's 7 TB / week*.
Awesome. Even given aggregate datasets from jobs, I think we're probably okay for space for at least 6 months. That's plenty of time evidence the value of analytics, and make a strong case for more disks. July is ~8 months away, so the timing lines up perfectly.
Everything else looks great. I think this leaves the cluster looking like this:
an01-an07 (7): ETL Workers
an08, an09 (2): Kafka Brokers
an10 (1): Primary NameNode
an11-an22 (12): Hadoop Workers & DataNodes
an23-an25 (3): Cluster-wide ZooKeepers
an26 (1): Monitoring (JMX & Ganglia)
an27 (1): Secondary NameNode, Hive Metastore (etc), Storm Nimbus
On Monday, 12 November 2012 at 9:08 a, Andrew Otto wrote: