Forge Home

confluent

contributions requested
This puppet module is used to install and configure the Confluent platform.
Project URLRSS FeedReport issues

28,364 downloads

3,681 latest version

4.5 quality score

Support the Puppet Community by contributing to this module

You are welcome to contribute to this module by suggesting new features, currency updates, or fixes. Every contribution is valuable to help ensure that the module remains compatible with the latest Puppet versions and continues to meet community needs. Complete the following steps:

  1. Review the module’s contribution guidelines and any licenses. Ensure that your planned contribution aligns with the author’s standards and any legal requirements.
  2. Fork the repository on GitHub, make changes on a branch of your fork, and submit a pull request. The pull request must clearly document your proposed change.

For questions about updating the module, contact the module’s author.

Version information

  • 4.0.55 (latest)
  • 4.0.54
  • 4.0.53
  • 4.0.52
  • 4.0.51
  • 4.0.49
  • 4.0.48
  • 4.0.47
  • 4.0.46
  • 3.3.45
  • 3.3.44
  • 3.3.43
  • 3.3.42
  • 3.3.40
  • 3.3.39
  • 3.3.38
  • 3.3.35
  • 3.3.34
  • 3.3.33
  • 3.3.32
  • 3.3.31
  • 3.3.30
  • 3.3.29
  • 3.3.28
  • 3.3.27
  • 3.2.26
  • 3.2.25
  • 3.2.24
  • 3.2.23
  • 3.2.22
  • 3.2.21
released Sep 29th 2019
This version is compatible with:
  • RedHat
    ,
    CentOS

Start using this module

  • r10k or Code Manager
  • Bolt
  • Manual installation
  • Direct download

Add this module to your Puppetfile:

mod 'jcustenborder-confluent', '4.0.55'
Learn more about managing modules with a Puppetfile

Add this module to your Bolt project:

bolt module add jcustenborder-confluent
Learn more about using this module with an existing project

Manually install this module globally with Puppet module tool:

puppet module install jcustenborder-confluent --version 4.0.55

Direct download is not typically how you would use a Puppet module to manage your infrastructure, but you may want to download the module in order to inspect the code.

Download

Documentation

jcustenborder/confluent — version 4.0.55 Sep 29th 2019

Introduction

This puppet module is used to install and configure the Confluent Platform. The documentation is available here.

Note on versioning

Puppet only allows version numbers in <major>.<minor>.<version> format. Due to this limitation the versioning schema matches the major and minor versions of the Confluent Platform. Meaning 3.2.1234 of the puppet module is for 3.2.x of the Confluent platform.

Usage

Zookeeper

Class installation

class{'confluent::zookeeper':
  zookeeper_id => '1',
  heap_size => '4000M'
}

Hiera Installation

include ::confluent::zookeeper

confluent::zookeeper::zookeeper_id: '1'
confluent::zookeeper::config:
  server.1:
    value: 'zookeeper-01.example.com:2888:3888'
  server.2:
    value: 'zookeeper-02.example.com:2888:3888'
  server.3:
    value: 'zookeeper-03.example.com:2888:3888'
confluent::zookeeper::heap_size: '4000M'

Kafka Broker

Class installation

class{'confluent::kafka::broker':
  broker_id => '1',
  zookeeper_connect => [
    'zookeeper-01:2181',
    'zookeeper-02:2181',
    'zookeeper-03:2181'
  ],
  heap_size => '4000M'
}

Heira installation

include ::confluent::kafka::broker

confluent::kafka::broker::broker_id: '1'
confluent::kafka::broker::heap_size: '4000M'
confluent::kafka::broker::zookeeper_connect:
  - 'zookeeper-01:2181',
  - 'zookeeper-02:2181',
  - 'zookeeper-03:2181'
confluent::kafka::broker::data_path: '/var/lib/kafka'

Kafka Connect

Distributed

Class Installation

class{'confluent::kafka::connect::distributed':
  heap_size => '4000M',
  bootstrap_servers => [
    'broker-01:9092',
    'broker-02:9092',
    'broker-03:9092'
  ],
  config => {
    'key.converter' => {
      'value' => 'io.confluent.connect.avro.AvroConverter'
    },
    'value.converter' => {
      'value' => 'io.confluent.connect.avro.AvroConverter'
    },
    'key.converter.schema.registry.url' => {
      'value' => 'http://schema-registry-01:8081'
    },
    'value.converter.schema.registry.url' => {
      'value' => 'http://schema-registry-01:8081'
    },
  },
}

Heira installation

include ::confluent::kafka::connect::distributed

 confluent::kafka::connect::distributed::heap_size: '4000M'
 confluent::kafka::connect::distributed::bootstrap_servers:
  - broker-01:9092
  - broker-02:9092
  - broker-03:9092
 confluent::kafka::connect::distributed::config:
   'key.converter':
     value: 'io.confluent.connect.avro.AvroConverter'
   'value.converter':
     value: 'io.confluent.connect.avro.AvroConverter'
   'key.converter.schema.registry.url':
     value: 'http://schema-registry-01.example.com:8081'
   'value.converter.schema.registry.url':
     value: 'http://schema-registry-01.example.com:8081'

Standalone

Class Installation

class{'confluent::kafka::connect::standalone':
  heap_size => '4000M',
  bootstrap_servers => [
    'broker-01:9092',
    'broker-02:9092',
    'broker-03:9092'
  ],
  config => {
    'key.converter' => {
      'value' => 'io.confluent.connect.avro.AvroConverter'
    },
    'value.converter' => {
      'value' => 'io.confluent.connect.avro.AvroConverter'
    },
    'key.converter.schema.registry.url' => {
      'value' => 'http://schema-registry-01:8081'
    },
    'value.converter.schema.registry.url' => {
      'value' => 'http://schema-registry-01:8081'
    },
  },
}

Heira installation

include ::confluent::kafka::connect::standalone

 confluent::kafka::connect::standalone::heap_size: '4000M'
 confluent::kafka::connect::standalone::bootstrap_servers:
  - broker-01:9092
  - broker-02:9092
  - broker-03:9092
 confluent::kafka::connect::standalone::config:
   'key.converter':
     value: 'io.confluent.connect.avro.AvroConverter'
   'value.converter':
     value: 'io.confluent.connect.avro.AvroConverter'
   'key.converter.schema.registry.url':
     value: 'http://schema-registry-01.example.com:8081'
   'value.converter.schema.registry.url':
     value: 'http://schema-registry-01.example.com:8081'

Schema Registry

Class installation

class {'confluent::schema::registry':
  heap_size => '1024M',
  kafkastore_connection_url => [
      'zookeeper-01:2181',
      'zookeeper-02:2181',
      'zookeeper-03:2181'
  ]
}

Hiera installation

include ::confluent::schema::registry

confluent::schema::registry::heap_size: '1024M'
confluent::schema::registry::kafkastore_connection_url:
  - 'zookeeper-01:2181'
  - 'zookeeper-02:2181'
  - 'zookeeper-03:2181'

Confluent Control Center

Class installation

class {'confluent::control::center':
  heap_size => '6g',
  zookeeper_connect => [
    'zookeeper-01:2181',
    'zookeeper-02:2181',
    'zookeeper-03:2181'
  ],
  bootstrap_servers => [
    'broker-01:9092',
    'broker-02:9092',
    'broker-03:9092'
  ],
  connect_cluster => [
    'kafka-connect-01.example.com:8083',
    'kafka-connect-02.example.com:8083',
    'kafka-connect-03.example.com:8083'
  ]
}

Hiera installation

include ::confluent::control::center

confluent::control::center::heap_size: '6g'
confluent::control::center::zookeeper_connect: 
  - 'zookeeper-01:2181'
  - 'zookeeper-02:2181'
  - 'zookeeper-03:2181'
confluent::control::center::bootstrap_servers: 
  - 'broker-01:9092'
  - 'broker-02:9092'
  - 'broker-03:9092'
confluent::control::center::connect_cluster: 
  - 'kafka-connect-01.example.com:8083'
  - 'kafka-connect-02.example.com:8083'
  - 'kafka-connect-03.example.com:8083'

Run tests

rake spec

Rebuild github pages

rake strings:gh_pages:update