How-to: Set Up an Apache Hadoop/Apache HBase Cluster on EC2 in (About) an Hour

Word (added July 8, 2013): The tips beneath is deprecated; we propose that you just seek advice from this post for present directions.

As of late we convey you one consumer’s revel in the use of Apache Whirr to spin up a CDH cluster within the cloud. This put up was once at first revealed here via George London (@rogueleaderr) in response to his private studies; he has graciously allowed us to convey it to you right here as smartly in a condensed shape. (Word: the configuration described right here is meant for studying/trying out functions most effective.)

I’m going to stroll you thru a (quite) easy set of steps that may get you up and operating MapReduce techniques on a cloud-based, six-node disbursed Apache Hadoop/Apache HBase cluster as speedy as conceivable. That is all in response to what I’ve picked up alone, so if of higher/quicker strategies, please let me know in feedback!

We’re going to be operating our cluster on Amazon EC2, and launching the cluster the use of Apache Whirr and configuring it the use of Cloudera Manager Free Edition.  Then we’ll run some elementary techniques I’ve posted on Github that may parse knowledge and cargo it into Apache HBase.

All in combination, this educational will take just a little over one hour and value about $10 in server prices.

Step 1: Get the Cluster Working

I’m going to suppose you have already got an Amazon Internet Products and services account (as it’s superior, and the elemental tier is unfastened.) In the event you don’t, go get one. Amazon’s instructions for buying began are beautiful transparent, or you’ll simply discover a information with Google. We gained’t in fact be interacting with the Amazon control console a lot, however you’ll want two items of knowledge, your AWS Get right of entry to Key ID and your AWS Secret Get right of entry to Key.

To seek out those, pass to You’ll be able to write those down, or higher but upload them in your shell startup script via doing:

$ echo “export AWS_ACCESS_KEY_ID=” > ~/.bashrc
$ echo “export AWS_SECRET_ACCESS_KEY=”your_key_here> > ~/.bashrc
$ exec $SHELL

You are going to additionally desire a safety certificates and personal key that may help you use the command-line equipment to have interaction with AWS. From the AWS Control Console pass to Account > Safety Credentials > Get right of entry to Credentials, make a choice the “X.509 Certificate” tab and click on on Create a brand new Certificates. Obtain and save this someplace secure (e.g. ~/.ec2)

Then do:

$ export EC2_PRIVATE_KEY=~/.ec2/.pem
$ export EC2_CERT=~/.ssh/.pem

In spite of everything, you’ll desire a other key to log into your servers the use of SSH. To create that, do:

$ mkdir ~/.ec2
$ ec2uploadkeypair area useast1 hadoop | sed 1d > ~/.ec2/hadoop
$ chmod 600 ~/.ec2/hadoop
(to lock down the permissions on the key so that SSH will agree to use it)

You might have the choice of manually growing a number of EC2 nodes, however that’s a ache. As an alternative, we’re going to make use of Whirr, which is particularly designed to permit push-button setup of clusters within the cloud.

To make use of Whirr, we’re going to wish to create one node manually, which we’re going to use as our “regulate middle.” I’m assuming you have got the EC2 command-line equipment put in (if no longer, pass here and observe instructions).

We’re going to create an occasion operating Ubuntu 10.04 (it’s previous, however the entire equipment we want run on it in solid style), and release it within the USA-East area. You’ll be able to in finding AMIs for different Ubuntu variations and areas here.

So, do:

$ ec2runoccasions ami1db20274 okay “hadoop”

This creates an EC2 occasion the use of a minimum Ubuntu picture, with the SSH key “hadoop_tutorial” that we created a second in the past. The command will produce a number of details about your occasion. Search for the “occasion identity” that begins with i- , then do:

$ ec2describeoccasion [ino matter]

This may occasionally inform you the IP of your new occasion (it’ll get started ec2-). Now we’re going to remotely log in to that server.

$ ssh i ~/.ec2/hadoop

Now we’re in! This server is most effective going to run two techniques, Whirr and the Cloudera Supervisor. First we’ll set up Whirr.  Discover a reflect at (, then obtain to your own home listing the use of wget:

$ wget

Untar and unzip:

$ tar xvf whirr0.8.0.tar.gz
$ cd whirr0.8.0

Whirr will release clusters for you in EC2 in keeping with a “homes” dossier you go it. It’s in fact reasonably tough and lets in a large number of customization (and can be utilized with non-Amazon cloud suppliers) otherwise you to arrange difficult servers the use of Chef scripts. However for our functions, we’ll stay it easy.

Create a dossier referred to as

$ nano

And provides it those contents:

whirr.occasiontemplates=6 noop

This may occasionally release a cluster of six unconfigured “massive” EC2 occasions. (Whirr refused to create small or medium occasions for me. Please let me know in feedback if you know the way to do this.)

Prior to we will use Whirr, we’re going to wish to set up Java, so do:

$ sudo aptget replace
$ sudo aptget set up openjdk6jreheadless

Subsequent we wish to create that SSH key that may let our regulate node log into to our cluster. 

$ sshkeygen t rsa P

And hit [enter] on the advised.

Now we’re able to release!

$ bin/whirr releasecluster config

This may occasionally produce a number of output and finish with instructions to SSH into your servers.

We’re going to wish those IPs for your next step, so reproduction and paste those strains into a brand new dossier:

$ nano hosts.txt

Then use this bit of standard expression magic to create a dossier with simply the IP’s:

$ sed rn “/|.*@(.*)’| s/.*@(.*)’//1/p” hosts.txt >> ips.txt

Step 2: Configure the Cluster

Out of your Regulate Node, obtain Cloudera Supervisor; we can set up the Loose Version, which can be utilized for as much as 50 nodes:

$ wget

Then set up it:

$ sudo chmod +x clouderasupervisorinstaller.bin
$ sudo ./clouderasupervisorinstaller.bin

This may occasionally pop up an excessive inexperienced set up wizard; simply hit “sure” to the whole thing.

Cloudera Supervisor works poorly with textual browsers like Lynx. (It has an API, however we gained’t quilt that right here.) Happily, we will get entry to the internet interface from our pc via having a look up the general public DNS deal with we used to log in to our regulate node, and appending “:7180” to the tip in our internet browser.

First, you want to inform Amazon to open that port. The chief additionally wishes a lovely ridiculously lengthy checklist of open ports to paintings, so we’re simply going to inform Amazon to open all TCP ports. That’s no longer nice for safety, so you’ll upload the person ports should you care sufficient (lists here):

$ ec2authorize default P tcp p 065535 o “jclouds#whirrly”
$ ec2authorize default P tcp p 7180 o
$ ec2authorize default P udp p 065535 o “jclouds#whirrly”
$ ec2authorize default P icmp t 1:1 o “jclouds#whirrly”

Then fan the flames of Chrome and discuss with .

Log in with the default credentials consumer: “admin” go: “admin”

Click on “simply set up the unfastened version”, “proceed”, then “continue” in tiny textual content on the backside proper of the registration display.

Now return to that ips.txt dossier we created within the ultimate section and replica the checklist of IPs. Previous them into the field at the subsequent display, and click on “seek”, then “set up CDH on decided on hosts.”

Subsequent the executive wishes credentials that’ll permit it to log into the nodes within the cluster to set them up. You want to present it a SSH key, however that key’s at the server and will’t be immediately accessed from you pc. So you want to replicate it in your pc.

$ scp r i ~/.ec2/hadoop_tutorial.pem ~/Downloads/hadoop_tutorial

(“scp” is a program that securely copies recordsdata thru ssh, and the –r flag will reproduction a listing.)

Now you’ll give the executive the username “huser”, and the SSH keys you simply downloaded:

Click on “get started set up,” then “good enough” to log in without a passphrase. Now look ahead to some time as CDH is put in on each and every node.

Subsequent, Cloudera Supervisor will investigate cross-check the hosts and problems some warnings however simply click on “proceed.” Then it’ll ask you which ones services and products you need to start out – make a selection “customized” after which make a choice Zookeeper, HDFS, HBase, and MapReduce.

Click on “proceed” at the “evaluate configuration adjustments” web page, then wait as the executive begins your services and products.

Click on “proceed” a pair extra instances when induced, and now you’ve were given a functioning cluster.

Step 3: Do One thing

To make use of your cluster, you want to SSH login to probably the most nodes. Pop open the “hosts.txt” dossier we made previous, grasp any of the strains, and paste it into the terminal.

$ ssh i /house/ubuntu/.ssh/id_rsa o “UserKnownHostsFile /dev/null” /
o StrictHostKeyChecking=no huser@

In the event you already know the way to make use of Hadoop and HBase, you then’re all accomplished. Your cluster is excellent to move. In the event you don’t, right here’s a temporary evaluation:

The elemental Hadoop workflow is to run a “process” that reads some knowledge from HDFS, “maps” some serve as onto that knowledge to procedure it, “reduces” the consequences again to a unmarried set of information, after which retail outlets the consequences again to HDFS. You’ll be able to additionally use HBase because the enter and/or output in your process.

You’ll be able to engage with HDFS immediately from the terminal thru instructions beginning “hadoop fs”. In CDH, Cloudera’s open-source Hadoop distro, you want to be logged in because the “hdfs” consumer to govern HDFS, so let’s log in as hdfs, create a customers listing for ourselves, then create an enter listing to retailer knowledge.

$ sudo su hdfs
$ hadoop fs mkdir p /consumer/hdfs/enter

You’ll be able to checklist the contents of HDFS via typing:

$ hadoop fs ls R /consumer

To run a program the use of MapReduce, you have got two choices. You’ll be able to both:

  • Write a program in Java the use of the MapReduce API and package deal it as a JAR
  • Use Hadoop Streaming, which lets you write your mapper and reducer”scripts in no matter language you need and transmit knowledge between phases via studying/writing to StdOut.

In the event you’re used to scripting languages like Python or Ruby and simply wish to crank thru some knowledge, Hadoop Streaming is superb (particularly since you’ll upload extra nodes to triumph over the relative CPU slowness of a better stage language). However interacting programmatically with HBase is so much more straightforward thru Java. (Interacting with HBase is hard however no longer not possible with Python. There’s a package deal referred to as “Happybase” which helps you to engage “pythonically” with HBase; the issue is that you must run a unique provider referred to as Thrift on each and every server to translate the Python directions into Java, or else transmit all your requests over the twine to a server on one node, which I guess will closely degrade efficiency. Cloudera Supervisor won’t arrange Thrift for you, even though you should do it via hand or the use of Whirr+Chef.) So I’ll supply a snappy instance of Hadoop streaming after which a extra prolonged HBase instance the use of Java.

Now, grasp my instance code repo off Github. We’ll want git.

(In the event you’re nonetheless logged in as hdfs, do “go out” again to “huser” since hdfs doesn’t have sudo privileges via default.)

$ sudo aptget set up y gitcore
$ sudo su hdfs
$ git clone $ cd Hadoop_Tutorial/hadoop_tutorial

Cloudera Supervisor gained’t inform the nodes the place to search out the configuration recordsdata it must run (i.e. “set the classpath”), so let’s do this now:

$ export HADOOP_CLASSPATH=/and so on/hbase/conf.cloudera.hbase1/:/and so on/hadoop/conf.cloudera.mapreduce1/:/and so on/hadoop/conf.cloudera.hdfs1/

Hadoop Streaming

Michael Noll has a excellent educational on Hadoop streaming with Python here. I’ve stolen the code and put it in Github for you, with the intention to get going. Load some pattern knowledge into hdfs:

$ hadoop fs copyFromLocal knowledge/sample_rdf.nt enter/sample_rdf.nt
$ hadoop fs ls R
(to see that the knowledge was once copied)

Now let’s Hadoop:

$ hadoop jar /usr/lib/hadoop0.20mapreduce/contrib/streaming/hadoopstreaming2.0.0mr1cdh4.0.1.jar /
dossier python/ mapper python/ /
dossier python/ reducer python/ /
enter /consumer/hdfs/enter/sample_rdf.nt output /consumer/hdfs/output/1

That’s a large honking commentary, however what it’s doing is telling Hadoop (which Cloudera Supervisor installs in /usr/lib/hadoop-0.20-mapreduce) to execute the “streaming” jar, to make use of the mapper and reducer “” and “”, passing the ones precise script recordsdata alongside to the entire nodes, telling it to perform at the sample_rdf.nt dossier, and to retailer the output within the (routinely created) output/1/ folder.

Let that run for a couple of mins, then verify that it labored via having a look on the knowledge:

$ hadoop fs cat /consumer/hdfs/output/1/section00000

That’s Hadoop Streaming in a nutshell. You’ll be able to execute no matter code you need in your mappers/reducers (e.g. Ruby and even shell instructions like “cat”. If you wish to use non-standardlib Python applications – e.g. “rdflib” for in fact parsing the RDF – you want to zip the applications and go the ones recordsdata to hadoop streaming the use of -file [].)

Hadoop/HBase API

If you wish to program immediately into Hadoop and HBase, you’ll do this the use of Java. The vital Java code can also be beautiful intimidating and verbose, however it’s somewhat easy if you get the grasp of it.

The Github repo we downloaded in Step Three incorporates some instance code that are supposed to simply run should you’ve adopted this information sparsely, and you’ll incrementally regulate that code in your personal functions. (The elemental code is customized from the code examples in Lars George’s HBase, the Definitive Guide. The entire authentic code can also be discovered here.  That code has its personal license, however my marginal adjustments are launched into the general public area.)

All you want to run the code is Maven. Snatch that:

(In the event you’re logged in as consumer “hdfs”, kind “go out” till you get again to huser. Or give hdfs sudo privileges with “visudo” if you know the way.)

$ sudo aptget set up maven2
$ sudo su hdfs $ cd Hadoop_Tutorial/hadoop_tutorial

While you run Hadoop jobs from the command line, Hadoop is actually transport your code over the twine to each and every of the nodes to be run in the neighborhood. So you want to wrap your code up right into a JAR dossier that incorporates your code and the entire dependencies. (There are different ways to package or transmit your code however I believe totally self-contained “fats jars” are the very best. You’ll be able to make those the use of the “colour” plugin which is incorporated within the instance challenge. )

Construct the jar dossier via typing:

$ export JAVA_HOME=/usr/lib/jvm/j2sdk1.6oracle/
(to inform maven the place java lives)
$ mvn package deal

That can take an irritatingly very long time (perhaps 20+ mins) as Maven downloads the entire dependencies, however it calls for no supervision.

(In the event you’re curious, you’ll take a look at the code with a textual content editor at /house/customers/hdfs/Hadoop_Tutorial/hadoop_tutorial/src/primary/java/com/tumblr/rogueleaderr/hadoop_tutorial/ There’s so much happening there, however I’ve attempted to make it clearer by the use of feedback.

Now we will in fact run our process:

$  cd /var/lib/hdfs/Hadoop_Tutorial/hadoop_educational
$ hadoop jar goal/uberhadoop_tutorial0.0.1SNAPSHOT.jar com.tumblr.rogueleaderr.hadoop_tutorial.HBaseMapReduceExample

In the event you get a number of connection mistakes, be certain your classpath is ready as it should be via doing:

$ export HADOOP_CLASSPATH=/and so on/hbase/conf.cloudera.hbase1/:/and so on/hadoop/conf.cloudera.mapreduce1/:/and so on/hadoop/conf.cloudera.hdfs1/

Verify that it labored via opening up the hbase commandline shell:

$ hbase shell
hbase(primary):001:0> scan “parsed_lines”

In the event you see a complete bunch of strains of information, then – congratulations! You’ve simply parsed RDF knowledge the use of a six-node Hadoop Cluster, and saved the leads to HBase!

Subsequent Steps

In the event you’re planning to do severe paintings with Hadoop and HBase, simply purchase the books:

The respectable tutorials for WhirrHadoop, and HBase are k, however beautiful intimidating for novices.

Past that, you must be capable of Google up some excellent tutorials.

Leave a Reply

Your email address will not be published. Required fields are marked *