Forge Home

Modules

Resources

PuppetEcosystemPuppet CommunityUpdates

kafka

Puppet module for Kafka
Project URLRSS FeedReport issues

366,425 downloads

10,855 latest version

5.0 quality score

We run a couple of automated
scans to help you access a
module's quality. Each module is
given a score based on how well
the author has formatted their
code and documentation and
modules are also checked for
malware using VirusTotal.

Please note, the information below
is for guidance only and neither of
these methods should be considered
an endorsement by Puppet.

Version information

  • 8.0.0 (latest)
  • 7.0.0
  • 6.0.0
  • 5.3.0
  • 5.2.0
  • 5.1.0
  • 5.0.1
  • 5.0.0
  • 4.0.0
  • 3.2.0
  • 3.1.0
  • 3.0.0
  • 2.3.0
  • 2.2.0
  • 2.1.0
  • 2.0.0
  • 1.0.3
  • 1.0.2
  • 1.0.1
released Oct 17th 2017
This version is compatible with:
  • Puppet Enterprise 2018.1.x, 2017.3.x, 2017.2.x, 2017.1.x, 2016.5.x, 2016.4.x
  • Puppet >= 4.7.1 < 6.0.0
  • , , ,

Start using this module

  • r10k or Code Manager
  • Bolt
  • Manual installation
  • Direct download

Add this module to your Puppetfile:

mod 'puppet-kafka', '4.0.0'
Learn more about managing modules with a Puppetfile

Add this module to your Bolt project:

bolt module add puppet-kafka
Learn more about using this module with an existing project

Manually install this module globally with Puppet module tool:

puppet module install puppet-kafka --version 4.0.0

Direct download is not typically how you would use a Puppet module to manage your infrastructure, but you may want to download the module in order to inspect the code.

Download
Tags: kafka, pubsub

Documentation

puppet/kafka — version 4.0.0 Oct 17th 2017

Kafka module for Puppet

Build Status Code Coverage Puppet Forge Puppet Forge - downloads Puppet Forge - endorsement Puppet Forge - scores

Table of Contents

  1. Overview
  2. Module Description - What the module does and why it is useful
  3. Setup - The basics of getting started with kafka
  4. Usage - Configuration options and additional functionality
  5. Reference - An under-the-hood peek at what the module is doing and how
  6. Limitations - OS compatibility, etc.
  7. Development - Guide for contributing to the module

Overview

The kafka module for managing the installation and configuration of Apache Kafka

Module Description

The Kafka module for managing the installation and configuration of Apache Kafka: it's brokers, producers and consumers.

Setup

What kafka affects

Installs the Kafka package and creates a new service.

Beginning with Kafka

To successfully install Kafka using this module you need to have Apache Zookeeper already running at localhost:2181. You can specify another Zookeeper host:port configuration using the config hash of the kafka:broker class.

The default configuration installs Kafka 0.9.0.1 binaries with Scala 2.11

  class { 'kafka': }

If you want a Kafka broker server that connects to zookeeper listening on port 2181:

  class { 'kafka::broker':
    config => { 'broker.id' => '0', 'zookeeper.connect' => 'localhost:2181' }
  }

Usage

You can specify different Kafka binaries packages versions to install. Please take a look at the different Scala and Kafka versions combinations at the Apache Kafka Website

Installing Kafka version 0.9.0.1 with scala 2.11

We first install the binary package with:

  class { 'kafka':
    version => '0.9.0.1',
    scala_version => '2.11'
  }

Then we set a minimal Kafka broker configuration with:

  class { 'kafka::broker':
    config => { 'broker.id' => '0', 'zookeeper.connect' => 'localhost:2181' }
  }

Classes and Defined Types

Class: kafka

One of the primary classes of the kafka module. This class will install the kafka binaries

Parameters within kafka:

version

The version of kafka that should be installed.

scala_version

The scala version what kafka was built with.

install_dir

The directory to install kafka to.

mirror_url

The url where the kafka is downloaded from.

install_java

Install java if it's not already installed.

package_dir

The directory to install kafka.

package_name

Package name, when installing kafka from a package.

package_ensure

Package version (or 'present', 'absent', 'latest'), when installing kafka from a package.

group_id

Create kafka group with this ID

user_id

Create kafka user with this ID

user

User to install kafka as. Defaults to the kafka user.

group

Group to install kafka as. Defaults to the kafka group.

config_dir

Directory for kafka config files. Defaults to /opt/kafka/config.

log_dir

Directory for kafka log files. Defaults to /var/log/kafka.

Class: kafka::broker

One of the primary classes of the kafka module. This class will install a kafka broker.

Parameters within kafka::broker:

version

The version of kafka that should be installed.

scala_version

The scala version what kafka was built with.

install_dir

The directory to install kafka to.

mirror_url

The url where the kafka is downloaded from.

config

A hash of the configuration options. All values are used in the server.properties file directly.

install_java

Install java if it's not already installed.

service_ensure

Sets the ensure state of the broker service to stopped or running.

service_install

Install the init.d service.

service_restart

Whether the configuration files should trigger a service restart

package_dir

The directory to install kafka.

Class: kafka::consumer

One of the primary classes of the kafka module. This class will install a kafka consumer.

Parameters within kafka::consumer:

version

The version of kafka that should be installed.

scala_version

The scala version that kafka was built with.

install_dir

The directory to install kafka to.

mirror_url

The url where the kafka is downloaded from.

install_java

Install java if it's not already installed.

package_dir

The directory to install kafka.

Class: kafka::mirror

One of the primary classes of the kafka module. This class will install a kafka mirror.

Parameters within kafka::mirror:

version

The version of kafka that should be installed.

scala_version

The scala version that kafka was built with.

install_dir

The directory to install kafka to.

mirror_url

The url where the kafka is downloaded from.

install_java

Install java if it's not already installed.

package_dir

The directory to install kafka.

Class: kafka::producer

One of the primary classes of the kafka module. This class will install a kafka producer.

Parameters within kafka::producer:

version

The version of kafka that should be installed.

scala_version

The scala version that kafka was built with.

install_dir

The directory to install kafka to.

mirror_url

The url where the kafka is downloaded from.

install_java

Install java if it's not already installed.

package_dir

The directory to install kafka.

Define: kafka::topic

Defined type that creates Kafka topics.

Reference

Classes

Public Classes

  • kafka: Guides the basic installation of kafka binaries
  • kafka::broker: Guides the basic installation of a kafka broker
  • kafka::consumer: Guides the basic installation of a kafka consumer
  • kafka::mirror: Guides the basic installation of a kafka mirror
  • kafka::producer: Guides the basic installation of a kafka producer

Private Classes

  • [kafka::broker::config] Manages all the default configuration of the kafka broker application
  • [kafka::broker::install] Manages the installation of the kafka packages
  • [kafka::broker::service] Manages the kafka server service
  • [kafka::consumer::config] Manages all the default configuration of the kafka consumer application
  • [kafka::consumer::install] Manages the installation of the kafka packages
  • [kafka::consumer::service] Manages the kafka server service
  • [kafka::mirror::config] Manages all the default configuration of the kafka mirror application
  • [kafka::mirror::install] Manages the installation of the kafka packages
  • [kafka::mirror::service] Manages the kafka server service
  • [kafka::producer::config] Manages all the default configuration of the kafka producer application
  • [kafka::producer::install] Manages the installation of the kafka packages
  • [kafka::producer::service] Manages the kafka server service

Defined Types

  • kafka::topic Create kafka topics

Limitations

This module is tested on the following platforms:

  • CentOS 5
  • CentOS 6
  • Ubuntu 12.04
  • Ubuntu 14.04

It is tested with the OSS version of Puppet (>= 4.7.0) only.

Development

Contributing

Please read CONTRIBUTING.md for full details on contributing to this project.