confluent

This puppet module is used to install and configure the Confluent platform.

Jeremy Custenborder

jcustenborder

21,907 downloads

2,662 latest version

4.5 quality score

Version information

  • 4.0.55 (latest)
  • 4.0.54
  • 4.0.53
  • 4.0.52
  • 4.0.51
  • 4.0.49
  • 4.0.48
  • 4.0.47
  • 4.0.46
  • 3.3.45
  • 3.3.44
  • 3.3.43
  • 3.3.42
  • 3.3.40
  • 3.3.39
  • 3.3.38
  • 3.3.35
  • 3.3.34
  • 3.3.33
  • 3.3.32
  • 3.3.31
  • 3.3.30
  • 3.3.29
  • 3.3.28
  • 3.3.27
  • 3.2.26
  • 3.2.25
  • 3.2.24
  • 3.2.23
  • 3.2.22
  • 3.2.21
released Sep 29th 2019
This version is compatible with:
  • RedHat
    ,
    CentOS

Start using this module

Documentation

jcustenborder/confluent — version 4.0.55 Sep 29th 2019

Introduction

This puppet module is used to install and configure the Confluent Platform. The documentation is available here.

Note on versioning

Puppet only allows version numbers in <major>.<minor>.<version> format. Due to this limitation the versioning schema matches the major and minor versions of the Confluent Platform. Meaning 3.2.1234 of the puppet module is for 3.2.x of the Confluent platform.

Usage

Zookeeper

Class installation

class{'confluent::zookeeper':
  zookeeper_id => '1',
  heap_size => '4000M'
}

Hiera Installation

include ::confluent::zookeeper
confluent::zookeeper::zookeeper_id: '1'
confluent::zookeeper::config:
  server.1:
    value: 'zookeeper-01.example.com:2888:3888'
  server.2:
    value: 'zookeeper-02.example.com:2888:3888'
  server.3:
    value: 'zookeeper-03.example.com:2888:3888'
confluent::zookeeper::heap_size: '4000M'

Kafka Broker

Class installation

class{'confluent::kafka::broker':
  broker_id => '1',
  zookeeper_connect => [
    'zookeeper-01:2181',
    'zookeeper-02:2181',
    'zookeeper-03:2181'
  ],
  heap_size => '4000M'
}

Heira installation

include ::confluent::kafka::broker
confluent::kafka::broker::broker_id: '1'
confluent::kafka::broker::heap_size: '4000M'
confluent::kafka::broker::zookeeper_connect:
  - 'zookeeper-01:2181',
  - 'zookeeper-02:2181',
  - 'zookeeper-03:2181'
confluent::kafka::broker::data_path: '/var/lib/kafka'

Kafka Connect

Distributed

Class Installation

class{'confluent::kafka::connect::distributed':
  heap_size => '4000M',
  bootstrap_servers => [
    'broker-01:9092',
    'broker-02:9092',
    'broker-03:9092'
  ],
  config => {
    'key.converter' => {
      'value' => 'io.confluent.connect.avro.AvroConverter'
    },
    'value.converter' => {
      'value' => 'io.confluent.connect.avro.AvroConverter'
    },
    'key.converter.schema.registry.url' => {
      'value' => 'http://schema-registry-01:8081'
    },
    'value.converter.schema.registry.url' => {
      'value' => 'http://schema-registry-01:8081'
    },
  },
}

Heira installation

include ::confluent::kafka::connect::distributed
 confluent::kafka::connect::distributed::heap_size: '4000M'
 confluent::kafka::connect::distributed::bootstrap_servers:
  - broker-01:9092
  - broker-02:9092
  - broker-03:9092
 confluent::kafka::connect::distributed::config:
   'key.converter':
     value: 'io.confluent.connect.avro.AvroConverter'
   'value.converter':
     value: 'io.confluent.connect.avro.AvroConverter'
   'key.converter.schema.registry.url':
     value: 'http://schema-registry-01.example.com:8081'
   'value.converter.schema.registry.url':
     value: 'http://schema-registry-01.example.com:8081'

Standalone

Class Installation

class{'confluent::kafka::connect::standalone':
  heap_size => '4000M',
  bootstrap_servers => [
    'broker-01:9092',
    'broker-02:9092',
    'broker-03:9092'
  ],
  config => {
    'key.converter' => {
      'value' => 'io.confluent.connect.avro.AvroConverter'
    },
    'value.converter' => {
      'value' => 'io.confluent.connect.avro.AvroConverter'
    },
    'key.converter.schema.registry.url' => {
      'value' => 'http://schema-registry-01:8081'
    },
    'value.converter.schema.registry.url' => {
      'value' => 'http://schema-registry-01:8081'
    },
  },
}

Heira installation

include ::confluent::kafka::connect::standalone
 confluent::kafka::connect::standalone::heap_size: '4000M'
 confluent::kafka::connect::standalone::bootstrap_servers:
  - broker-01:9092
  - broker-02:9092
  - broker-03:9092
 confluent::kafka::connect::standalone::config:
   'key.converter':
     value: 'io.confluent.connect.avro.AvroConverter'
   'value.converter':
     value: 'io.confluent.connect.avro.AvroConverter'
   'key.converter.schema.registry.url':
     value: 'http://schema-registry-01.example.com:8081'
   'value.converter.schema.registry.url':
     value: 'http://schema-registry-01.example.com:8081'

Schema Registry

Class installation

class {'confluent::schema::registry':
  heap_size => '1024M',
  kafkastore_connection_url => [
      'zookeeper-01:2181',
      'zookeeper-02:2181',
      'zookeeper-03:2181'
  ]
}

Hiera installation

include ::confluent::schema::registry
confluent::schema::registry::heap_size: '1024M'
confluent::schema::registry::kafkastore_connection_url:
  - 'zookeeper-01:2181'
  - 'zookeeper-02:2181'
  - 'zookeeper-03:2181'

Confluent Control Center

Class installation

class {'confluent::control::center':
  heap_size => '6g',
  zookeeper_connect => [
    'zookeeper-01:2181',
    'zookeeper-02:2181',
    'zookeeper-03:2181'
  ],
  bootstrap_servers => [
    'broker-01:9092',
    'broker-02:9092',
    'broker-03:9092'
  ],
  connect_cluster => [
    'kafka-connect-01.example.com:8083',
    'kafka-connect-02.example.com:8083',
    'kafka-connect-03.example.com:8083'
  ]
}

Hiera installation

include ::confluent::control::center
confluent::control::center::heap_size: '6g'
confluent::control::center::zookeeper_connect: 
  - 'zookeeper-01:2181'
  - 'zookeeper-02:2181'
  - 'zookeeper-03:2181'
confluent::control::center::bootstrap_servers: 
  - 'broker-01:9092'
  - 'broker-02:9092'
  - 'broker-03:9092'
confluent::control::center::connect_cluster: 
  - 'kafka-connect-01.example.com:8083'
  - 'kafka-connect-02.example.com:8083'
  - 'kafka-connect-03.example.com:8083'

Run tests

rake spec

Rebuild github pages

rake strings:gh_pages:update