hadoop

Install Hadoop MapReduce Next Generation.

viirya

viirya

13,614 downloads

9,335 latest version

1.5 quality score

Version information

  • 0.0.18 (latest)
  • 0.0.17
  • 0.0.16
  • 0.0.15
  • 0.0.14
  • 0.0.13
  • 0.0.12
  • 0.0.11
  • 0.0.10
  • 0.0.9
  • 0.0.8
  • 0.0.7
  • 0.0.6
  • 0.0.5
  • 0.0.4
  • 0.0.3
  • 0.0.2
  • 0.0.1
released May 28th 2014

Start using this module

Tags: hadoop, yarn

Documentation

viirya/hadoop — version 0.0.18 May 28th 2014

Puppet module for deploying Hadoop MapReduce Next Generation on cluster

This module deploys Hadoop MapReduce Next Generation on a cluster of machines. It is tested on Apache Hadoop 2.2.0 under puppet agent/master environment. It is based on bcarpio/hadoop 0.0.3.

Usage:

Install this module on your puppet master node by:

sudo puppet install viirya-hadoop.

In site.pp, define:

node 'your hadoop slave nodes' {
    include java
    include hadoop::cluster::slave
}

node 'your hadoop master node' {
    include java
    include hadoop::cluster::master
}

For pseudo-distributed mode of Hadoop, put following codes in a .pp file (e.g. hadoop.pp).

include java
include hadoop::cluster::pseudomode

Then,

sudo puppet apply hadoop.pp

Hadoop Distribution:

This puppet module will automatically download Apache Hadoop distribution from one pre-defined Apache Mirror site. If you are like to use faster mirror site, please modify the URL in init.pp.

Parameters:

Some parameters are able to modify in params.pp. You should modify the parameters such as 'master', 'resourcemanager', 'dfs_slaves' and 'yarn_slaves' to reflect your hadoop cluster settings.

SSH keys:

Remember to generate your ssh keys and put the keys in files/ssh/.

Note: Since master deployment will run hadoop scripts to launch hadoop services on slaves nodes, please deploy hadoop slaves first. When all slaves are deployed, then deploy master node.

Kerberos:

Since 0.0.12 version, this puppet module supports Kerberos-secured Hadoop cluster. Kerberos should be installed first on your cluster. To generate necessary principles and keytab files for Hadoop serviers:

require hadoop::cluster::kerberos