As discussed in the TechOps meeting yesterday, the Analytics team is evaluating two different Hadoop distributions for the batch processing layer of the Kraken cluster: Cloudera Hadoop 4 (CDH4) and Datastax Enterprise (DSE). I'm going to try to describe from a very high level why we think DSE is purrrrrty cool, and why CDH4 sounds like a relative headache. I'll then ask the questions we hope to answer soon.
(Quick disclaimer: This is a totally biased and incomplete summary of DSE vs CDH4. I am highlighting weaknesses of CDH4 in order to illustrate why we are considering DSE. If we are allowed to use DSE, then we will have more work to do weighing pros and cons of both solutions.)
CDH4 is just a well packaged distribution of Hadoop and some of the most useful Hadoop tools. The core of Hadoop is MapReduce and the Hadoop Distributed Filesystem (HDFS). HDFS allows huge files to be written across any number of machines. MapReduce jobs can then be written to do parallel processing of these files.
Hadoop has a single NameNode that is responsible for managing file metadata in HDFS. This is a notorious single point of failure.
CDH4 attempts to solve this problem with the introduction of a Standby NameNode[1]. This requires that both the Active and Standby NameNodes share a filesystem (via NFS or something similar). The Standby nodes uses this to synchronize edits to its own file metadata namespace. If the Active NameNode goes down, the Standby can take over and *should* have the exact same metadata stored as the Active. Administrators have to take special care to ensure that only one of the two NameNodes is active at once, or metadata corruption could result. Read about Fencing[2] for more info.
Not only is the NameNode a SPOF, but it doesn't scale well when there are many files or many metadata operations. More recent Hadoop releases solve this providing 'HDFS Federation'[3]. Basically, this is just multiple NameNodes sharing the independent spaces of the same data nodes. This is all more configuration to get Hadoop to scale, something it is supposed to be good at out of the box.
All of the above also comes with lots of configuration to maintain and tweak to make work. DSE is supposed to solve these problems and be much easier to work with.
DSE is Hadoop without HDFS. Instead, Datastax has written an HDFS emulation layer on top of Cassandra (CFS). The huge benefit here is that from the clients viewpoint, everything works just like Hadoop[4]. MapReduce jobs and all of the fancy Hadoop tools still work. But there's no NameNode to deal with. In Cassandra all nodes are peers. This allows for a much more homogenous cluster and less configuration[5]. Cassandra automatically rebalances its data when a node goes down.
Ah but DSE has WMF related problems of its own! Most importantly, DSE is not 100% open source*. The core of DSE is open source components available under the Apache license (Hadoop, Cassandra, etc.). As far as I can tell, anything that is not an Apache project is proprietary. Datastax has a comparison[7] of its Community vs. Enterprise versions. This includes packaging and wrapper CLI tools, but most importantly the Cassandra File System (CFS) emulation layer. This piece is what makes DSE so attractive.
Also note that DSE is not free. It is for development purposes, but it has a license that we'd have to buy if we want to run it in production. However, Diederik knows one of the founders of Datastax who might give it to us at a discount, or perhaps for free.
Okokok. So I'm starting this thread in order to answer my big question: Are we allowed to use DSE even if it is not 100% open source? If the answer is an easy 'no', then our choice is easy: we will use CDH4. Is it possible to get an answer to this question before we go down the road of dedicating time to evaluating and learning DSE?
tl;dr: Datastax is cooler than Cloudera, but Datastax is not 100% open source. Can we still use it?
- otto + the Analytics Team
*much like Cloudera for Hadoop, DataStax publishes tested, stable distributions of Cassandra and ecosystem tools bundled together into one package. In addition to selling support, their Enterprise edition contains proprietary code that, among other things, provides HDFS emulation on top Cassandra. This code originated as Brisk[7], and was forked when the developers founded DataStax.
[1] https://ccp.cloudera.com/display/CDH4DOC/Introduction+to+Hadoop+High+Availab... [2] https://ccp.cloudera.com/display/FREE4DOC/Configuring+HDFS+High+Availability [3] http://hadoop.apache.org/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/Federatio... [4] http://www.datastax.com/faq#dse-2 [5] http://blog.milford.io/2011/04/why-i-am-very-excited-about-datastaxs-brisk/ [6] http://wiki.apache.org/cassandra/HadoopSupport [7] https://github.com/riptano/brisk , https://github.com/riptano/brisk-hadoop-common
Ah, if you are interested in reading more Datastax propaganda, I just found this. http://www.datastax.com/wp-content/uploads/2012/09/WP-DataStax-HDFSvsCFS.pdf
On Sep 25, 2012, at 10:37 AM, Andrew Otto otto@wikimedia.org wrote:
As discussed in the TechOps meeting yesterday, the Analytics team is evaluating two different Hadoop distributions for the batch processing layer of the Kraken cluster: Cloudera Hadoop 4 (CDH4) and Datastax Enterprise (DSE). I'm going to try to describe from a very high level why we think DSE is purrrrrty cool, and why CDH4 sounds like a relative headache. I'll then ask the questions we hope to answer soon.
(Quick disclaimer: This is a totally biased and incomplete summary of DSE vs CDH4. I am highlighting weaknesses of CDH4 in order to illustrate why we are considering DSE. If we are allowed to use DSE, then we will have more work to do weighing pros and cons of both solutions.)
CDH4 is just a well packaged distribution of Hadoop and some of the most useful Hadoop tools. The core of Hadoop is MapReduce and the Hadoop Distributed Filesystem (HDFS). HDFS allows huge files to be written across any number of machines. MapReduce jobs can then be written to do parallel processing of these files.
Hadoop has a single NameNode that is responsible for managing file metadata in HDFS. This is a notorious single point of failure.
CDH4 attempts to solve this problem with the introduction of a Standby NameNode[1]. This requires that both the Active and Standby NameNodes share a filesystem (via NFS or something similar). The Standby nodes uses this to synchronize edits to its own file metadata namespace. If the Active NameNode goes down, the Standby can take over and *should* have the exact same metadata stored as the Active. Administrators have to take special care to ensure that only one of the two NameNodes is active at once, or metadata corruption could result. Read about Fencing[2] for more info.
Not only is the NameNode a SPOF, but it doesn't scale well when there are many files or many metadata operations. More recent Hadoop releases solve this providing 'HDFS Federation'[3]. Basically, this is just multiple NameNodes sharing the independent spaces of the same data nodes. This is all more configuration to get Hadoop to scale, something it is supposed to be good at out of the box.
All of the above also comes with lots of configuration to maintain and tweak to make work. DSE is supposed to solve these problems and be much easier to work with.
DSE is Hadoop without HDFS. Instead, Datastax has written an HDFS emulation layer on top of Cassandra (CFS). The huge benefit here is that from the clients viewpoint, everything works just like Hadoop[4]. MapReduce jobs and all of the fancy Hadoop tools still work. But there's no NameNode to deal with. In Cassandra all nodes are peers. This allows for a much more homogenous cluster and less configuration[5]. Cassandra automatically rebalances its data when a node goes down.
Ah but DSE has WMF related problems of its own! Most importantly, DSE is not 100% open source*. The core of DSE is open source components available under the Apache license (Hadoop, Cassandra, etc.). As far as I can tell, anything that is not an Apache project is proprietary. Datastax has a comparison[7] of its Community vs. Enterprise versions. This includes packaging and wrapper CLI tools, but most importantly the Cassandra File System (CFS) emulation layer. This piece is what makes DSE so attractive.
Also note that DSE is not free. It is for development purposes, but it has a license that we'd have to buy if we want to run it in production. However, Diederik knows one of the founders of Datastax who might give it to us at a discount, or perhaps for free.
Okokok. So I'm starting this thread in order to answer my big question: Are we allowed to use DSE even if it is not 100% open source? If the answer is an easy 'no', then our choice is easy: we will use CDH4. Is it possible to get an answer to this question before we go down the road of dedicating time to evaluating and learning DSE?
tl;dr: Datastax is cooler than Cloudera, but Datastax is not 100% open source. Can we still use it?
- otto + the Analytics Team
*much like Cloudera for Hadoop, DataStax publishes tested, stable distributions of Cassandra and ecosystem tools bundled together into one package. In addition to selling support, their Enterprise edition contains proprietary code that, among other things, provides HDFS emulation on top Cassandra. This code originated as Brisk[7], and was forked when the developers founded DataStax.
[1] https://ccp.cloudera.com/display/CDH4DOC/Introduction+to+Hadoop+High+Availab... [2] https://ccp.cloudera.com/display/FREE4DOC/Configuring+HDFS+High+Availability [3] http://hadoop.apache.org/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/Federatio... [4] http://www.datastax.com/faq#dse-2 [5] http://blog.milford.io/2011/04/why-i-am-very-excited-about-datastaxs-brisk/ [6] http://wiki.apache.org/cassandra/HadoopSupport [7] https://github.com/riptano/brisk , https://github.com/riptano/brisk-hadoop-common
On Tue, Sep 25, 2012 at 10:37:36AM -0400, Andrew Otto wrote:
Okokok. So I'm starting this thread in order to answer my big question: Are we allowed to use DSE even if it is not 100% open source? If the answer is an easy 'no', then our choice is easy: we will use CDH4. Is it possible to get an answer to this question before we go down the road of dedicating time to evaluating and learning DSE?
Based on your choice of words ("allowed"), I think you're looking for some kind of authoritative/executive decision and I'm not sure who can give you that.
As far as my personal opinion goes, I'd hate us to see using so-called "Enterpise" editions and non-free (as in freedom) software. I realize we have a few exceptions already (incl. Oracle Java) but this is a slippery slope to be in. I strongly believe in free & open-source software and its values -- values similar to those of the free culture movement, which we're supposed to support.
I also enjoy the freedoms on a daily basis, incl. studying the code that we use to understand the software and the bugs I'm finding as well as fixing those bugs with no EULA restrictions or binaries to fiddle with. We're also interacting with the communities of said software, giving them feedback and patches and getting "support" for free, without needing friends or special company relations (or paid contracts). Yes, on a *daily* basis, I can think of numerous examples.
I strongly believe that we should support the free alternatives even if they suck more at the moment, possibly enhance them and contribute them back to the larger Hadoop & Cassandra communities.
As a final note and as I mentioned at the meeting yesterday, we currently have no infrastructure to support the use of proprietary software: our repositories (apt, puppet etc.) are public and we're effectively redistributing everything that we put there, which is going to class with a proprietary license. But that's probably a smaller, one-off cost of setting something up for that.
Best, Faidon
Aye, one real quick clarification:
I strongly believe that we should support the free alternatives even if they suck more at the moment, possibly enhance them and contribute them back to the larger Hadoop & Cassandra communities.
Using DSE doesn't prohibit us from doing that. DSE has a couple of closed pieces, most importantly CFS, which ties Hadoop and Cassandra together. We'd still be using open source versions of those.
On Sep 25, 2012, at 2:24 PM, Faidon Liambotis faidon@wikimedia.org wrote:
On Tue, Sep 25, 2012 at 10:37:36AM -0400, Andrew Otto wrote:
Okokok. So I'm starting this thread in order to answer my big question: Are we allowed to use DSE even if it is not 100% open source? If the answer is an easy 'no', then our choice is easy: we will use CDH4. Is it possible to get an answer to this question before we go down the road of dedicating time to evaluating and learning DSE?
Based on your choice of words ("allowed"), I think you're looking for some kind of authoritative/executive decision and I'm not sure who can give you that.
As far as my personal opinion goes, I'd hate us to see using so-called "Enterpise" editions and non-free (as in freedom) software. I realize we have a few exceptions already (incl. Oracle Java) but this is a slippery slope to be in. I strongly believe in free & open-source software and its values -- values similar to those of the free culture movement, which we're supposed to support.
I also enjoy the freedoms on a daily basis, incl. studying the code that we use to understand the software and the bugs I'm finding as well as fixing those bugs with no EULA restrictions or binaries to fiddle with. We're also interacting with the communities of said software, giving them feedback and patches and getting "support" for free, without needing friends or special company relations (or paid contracts). Yes, on a *daily* basis, I can think of numerous examples.
I strongly believe that we should support the free alternatives even if they suck more at the moment, possibly enhance them and contribute them back to the larger Hadoop & Cassandra communities.
As a final note and as I mentioned at the meeting yesterday, we currently have no infrastructure to support the use of proprietary software: our repositories (apt, puppet etc.) are public and we're effectively redistributing everything that we put there, which is going to class with a proprietary license. But that's probably a smaller, one-off cost of setting something up for that.
Best, Faidon
I don't think NameNode frailty is that big of an issue. Hadoop with HDFS has been used successfully for years at scales that dwarf what we are considering, from its inception at yahoo across 40k nodes, to at facebook today. There are multiple approaches to NameNode redundancy. It can effectively be an ops issue involving 2 or 3 servers utilizing old tested tools for making applications not designed for high availability, highly available. But Cloudera's out-of-the-box solution here looks reasonable at first glance, and there are other alternatives such as Facebook's open-source NameNode replacement.
With Facebook recently blogging "Whether your system has hundreds of nodes or thousands, HDFS is the most scalable and most reliable open-source distributed filesystem available." ( https://www.facebook.com/notes/facebook-engineering/under-the-hood-hadoop-di...) I'm inclined to give their opinion greater credibility than any marketing copy, in the absence of our own data.
Take the NameNode controversy out of the picture, and what is the pitch for DSE?
What data processing patterns are better handled by tools built on top of Hadoop than by native Cassandra queries? Are such tools being considered only because they are better supported and easier to use by analysts, or are they technically superior in some ways to a pure Cassandra solution? Does Cassandra's on-disk storage format always map well to the usage patterns of tools like Hive?
Wondering generically about the use of Cassandra as a distributed file-system replacement, would you consider it as an object store for media files? It seems like it isn't well suited for such a task, while HDFS is a potential option for replacing Swift and can be used for video streaming. This may seem irrelevant to Kraken, and yet the fact that HDFS can stream large objects while Cassandra cannot may impact the performance of data analysis tools designed for HDFS. There's also the (perhaps small) possibility that we will in fact replace Swift with HDFS. If that occurs and keeping an HDFS cluster available becomes a primary duty of the ops team, they should be able to take responsibility for a Kraken HDFS cluster as well.
-Asher
On Tue, Sep 25, 2012 at 7:37 AM, Andrew Otto otto@wikimedia.org wrote:
As discussed in the TechOps meeting yesterday, the Analytics team is evaluating two different Hadoop distributions for the batch processing layer of the Kraken cluster: Cloudera Hadoop 4 (CDH4) and Datastax Enterprise (DSE). I'm going to try to describe from a very high level why we think DSE is purrrrrty cool, and why CDH4 sounds like a relative headache. I'll then ask the questions we hope to answer soon.
(Quick disclaimer: This is a totally biased and incomplete summary of DSE vs CDH4. I am highlighting weaknesses of CDH4 in order to illustrate why we are considering DSE. If we are allowed to use DSE, then we will have more work to do weighing pros and cons of both solutions.)
CDH4 is just a well packaged distribution of Hadoop and some of the most useful Hadoop tools. The core of Hadoop is MapReduce and the Hadoop Distributed Filesystem (HDFS). HDFS allows huge files to be written across any number of machines. MapReduce jobs can then be written to do parallel processing of these files.
Hadoop has a single NameNode that is responsible for managing file metadata in HDFS. This is a notorious single point of failure.
CDH4 attempts to solve this problem with the introduction of a Standby NameNode[1]. This requires that both the Active and Standby NameNodes share a filesystem (via NFS or something similar). The Standby nodes uses this to synchronize edits to its own file metadata namespace. If the Active NameNode goes down, the Standby can take over and *should* have the exact same metadata stored as the Active. Administrators have to take special care to ensure that only one of the two NameNodes is active at once, or metadata corruption could result. Read about Fencing[2] for more info.
Not only is the NameNode a SPOF, but it doesn't scale well when there are many files or many metadata operations. More recent Hadoop releases solve this providing 'HDFS Federation'[3]. Basically, this is just multiple NameNodes sharing the independent spaces of the same data nodes. This is all more configuration to get Hadoop to scale, something it is supposed to be good at out of the box.
All of the above also comes with lots of configuration to maintain and tweak to make work. DSE is supposed to solve these problems and be much easier to work with.
DSE is Hadoop without HDFS. Instead, Datastax has written an HDFS emulation layer on top of Cassandra (CFS). The huge benefit here is that from the clients viewpoint, everything works just like Hadoop[4]. MapReduce jobs and all of the fancy Hadoop tools still work. But there's no NameNode to deal with. In Cassandra all nodes are peers. This allows for a much more homogenous cluster and less configuration[5]. Cassandra automatically rebalances its data when a node goes down.
Ah but DSE has WMF related problems of its own! Most importantly, DSE is not 100% open source*. The core of DSE is open source components available under the Apache license (Hadoop, Cassandra, etc.). As far as I can tell, anything that is not an Apache project is proprietary. Datastax has a comparison[7] of its Community vs. Enterprise versions. This includes packaging and wrapper CLI tools, but most importantly the Cassandra File System (CFS) emulation layer. This piece is what makes DSE so attractive.
Also note that DSE is not free. It is for development purposes, but it has a license that we'd have to buy if we want to run it in production. However, Diederik knows one of the founders of Datastax who might give it to us at a discount, or perhaps for free.
Okokok. So I'm starting this thread in order to answer my big question: Are we allowed to use DSE even if it is not 100% open source? If the answer is an easy 'no', then our choice is easy: we will use CDH4. Is it possible to get an answer to this question before we go down the road of dedicating time to evaluating and learning DSE?
tl;dr: Datastax is cooler than Cloudera, but Datastax is not 100% open source. Can we still use it?
- otto + the Analytics Team
*much like Cloudera for Hadoop, DataStax publishes tested, stable distributions of Cassandra and ecosystem tools bundled together into one package. In addition to selling support, their Enterprise edition contains proprietary code that, among other things, provides HDFS emulation on top Cassandra. This code originated as Brisk[7], and was forked when the developers founded DataStax.
[1] https://ccp.cloudera.com/display/CDH4DOC/Introduction+to+Hadoop+High+Availab... [2] https://ccp.cloudera.com/display/FREE4DOC/Configuring+HDFS+High+Availability [3] http://hadoop.apache.org/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/Federatio... [4] http://www.datastax.com/faq#dse-2 [5] http://blog.milford.io/2011/04/why-i-am-very-excited-about-datastaxs-brisk/ [6] http://wiki.apache.org/cassandra/HadoopSupport [7] https://github.com/riptano/brisk , https://github.com/riptano/brisk-hadoop-common
Ops mailing list Ops@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/ops
On Tuesday, September 25, 2012 at 3:25 PM, Asher Feldman wrote:
What data processing patterns are better handled by tools built on top of Hadoop than by native Cassandra queries?
Range scans. Cassandra has two data partitioner classes: RandomPartitioner and ByteOrderedPartitioner. The latter *does* allow you to partition data lexically by key bytes, but by going that route you now have to think about load balancing explicitly, and it's easy to make mistakes. You basically lose the cruise-control load balancing / node management benefits of a DHT like Cassandra. Because that's so, "DataStax strongly recommends against using the ordered partitioner", citing the following reasons:
- Sequential writes can cause hot spots - More administrative overhead to load balance the cluster - Uneven load balancing for multiple column families
More details here: http://www.datastax.com/docs/1.0/cluster_architecture/partitioning#byteorder...
HDFS is explicitly designed to help you reason about and manage data locality.
On Tue, Sep 25, 2012 at 11:37 PM, Ori Livneh ori.livneh@gmail.com wrote:
- Sequential writes can cause hot spots
- More administrative overhead to load balance the cluster
- Uneven load balancing for multiple column families
Can you say more about the planned usage/workload? What key would you be partitioning on? Sounds like maybe hbase is worth a look.
-Jeremy
On Tue, Sep 25, 2012 at 7:37 AM, Andrew Otto otto@wikimedia.org wrote:
tl;dr: Datastax is cooler than Cloudera, but Datastax is not 100% open source. Can we still use it?
Not the proprietary bits, no.
If it's just about using the open core part, that's less problematic, although I generally advise against buying into open core solutions because the incentive structures aren't aligned (they'll continually try to upsell you on the cool proprietary features).
Not the proprietary bits, no.
Cool! This is exactly why I wanted to ask this question. This is actually a bit of a relief! While I would love to try out DSE, CDH4 will almost certainly do the job. Having the choice made for me saves me a lot of work in evaluating and researching multiple options.
Thanks all!
On Sep 25, 2012, at 7:34 PM, Erik Moeller erik@wikimedia.org wrote:
On Tue, Sep 25, 2012 at 7:37 AM, Andrew Otto otto@wikimedia.org wrote:
tl;dr: Datastax is cooler than Cloudera, but Datastax is not 100% open source. Can we still use it?
Not the proprietary bits, no.
If it's just about using the open core part, that's less problematic, although I generally advise against buying into open core solutions because the incentive structures aren't aligned (they'll continually try to upsell you on the cool proprietary features).
-- Erik Möller VP of Engineering and Product Development, Wikimedia Foundation
Support Free Knowledge: https://wikimediafoundation.org/wiki/Donate
Ops mailing list Ops@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/ops
Oo, good news! From http://www.cloudera.com/blog/2012/10/cdh4-1-now-released/
Quorum based storage – Quorum-based Storage for HDFS provides the ability for HDFS to store its own NameNode edit logs, allowing you to run a highly available NameNode without external storage or custom fencing.
This one of the reasons we wanted to use DSE in the first place. I haven't looked into the details here, but now we should be able to run HA NameNode without NFS!
On Sep 26, 2012, at 2:44 PM, Andrew Otto otto@wikimedia.org wrote:
Not the proprietary bits, no.
Cool! This is exactly why I wanted to ask this question. This is actually a bit of a relief! While I would love to try out DSE, CDH4 will almost certainly do the job. Having the choice made for me saves me a lot of work in evaluating and researching multiple options.
Thanks all!
On Sep 25, 2012, at 7:34 PM, Erik Moeller erik@wikimedia.org wrote:
On Tue, Sep 25, 2012 at 7:37 AM, Andrew Otto otto@wikimedia.org wrote:
tl;dr: Datastax is cooler than Cloudera, but Datastax is not 100% open source. Can we still use it?
Not the proprietary bits, no.
If it's just about using the open core part, that's less problematic, although I generally advise against buying into open core solutions because the incentive structures aren't aligned (they'll continually try to upsell you on the cool proprietary features).
-- Erik Möller VP of Engineering and Product Development, Wikimedia Foundation
Support Free Knowledge: https://wikimediafoundation.org/wiki/Donate
Ops mailing list Ops@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/ops
On Sep 25, 2012, at 4:37 PM, Andrew Otto otto@wikimedia.org wrote:
As discussed in the TechOps meeting yesterday, the Analytics team is evaluating two different Hadoop distributions for the batch processing layer of the Kraken cluster: Cloudera Hadoop 4 (CDH4) and Datastax Enterprise (DSE). I'm going to try to describe from a very high level why we think DSE is purrrrrty cool, and why CDH4 sounds like a relative headache. I'll then ask the questions we hope to answer soon.
[snip]
Most of this has already been said, so I'll keep it short now: Using Free and Open Source Software is an important part of what we do. There are many cases of open source software in use in our cluster with non-free "Enterprise" versions out there, with more and sometimes arguably better features than in the "open (source) core" versions. We generally don't use those non-free versions, especially not when it's a conscious decision made as a team. Even purely on a technical level, I don't think we've ever regretted those decisions, whereas we're constantly trying to move away from the few exceptions that have slipped through the cracks in the past.
We only use non-free solutions where there is no realistic, reasonable open source alternative available at all. With Enterprise versions based on existing open source software that's almost never the case, and in this particular case, judging from your comments as well as others, it doesn't seem to apply either. It's perfectly possible to use a fully open source package (Cloudera Hadoop) for your purposes, and thus that's what you should do at Wikimedia.