Create EMR Cluster

description

This shows how to create an EMR (Elastic Map Reduce) cluster on AWS

Notes

• This module utilies AWS’s EMR Ruby client, there are some more options that that client allows use –options to pass these to the AWS client
• This module install the elastic-HPC Server so that user can interact with the server from outside.
• to save your credentials, so that you don’t pass them, check help of AWS Ruby Client help.
• to install a bioinformatics tool on the cluster, you need to use bootstrap script to install the tool as the cluster starts, here we show the bootstrap for crossbow an example

cd /home/hadoop
wget http://dl.dropbox.com/u/18402489/crossbow-1.1.2.zip
unzip crossbow-1.1.2.zip
chmod -R 777 crossbow-1.1.2/*
mv crossbow-1.1.2 crossbow
cd crossbow/bin/
wget http://ftp-private.ncbi.nlm.nih.gov/sra/sdk/2.1.9/sratoolkit.2.1.9-centos_linux64.tar.gz
tar xzvf sratoolkit.2.1.9-centos_linux64.tar.gz
sudo cp sratoolkit.2.1.9-centos_linux64/fastq-dump.2.1.9 /usr/bin/fastq-dump

Command-line Interface

>>> ,.EHPC-EMR --create OPTIONS
All parameters should be in format parameter=value
--num-instances, -n         number of instance in the cluster, default 1
--instance-type, -t         type of cluster instances, default m1.small
--install, -i               comma seprated list of bootstrap scripts to run
--region, -r                region to start the cluster in,e.g. eu-west-1, us-east-1, default us-east-1
--options                   pass other options to AWS Client
AWS Credentials:
--acessKey, -a              access to start the cluster with
--privateKey, -p            private key  to start tne cluster with
--key-pair, -kp             name of the keypair with start the cluster with
--key-pair-file, -kf        path of the keypair file

Example 1

Start a EMR Cluster of 4 nodes of type m1.large in us-east-1

./EHPC-EMR --create -n=4 -t=m1.large

Output

DNS  of new Cluster

Example 2

Start a EMR Cluster of 3 nodes where the main node of type m1.large and other 2 is of type m1.xlarge

./EHPC-EMR --create -n=3 -t= --options="--master-instance-type m1.large --slave-instance-type m1.xlarge

Output

DNS  of new Cluster