Forge Home

Modules

Resources

PuppetEcosystemPuppet CommunityUpdates

elkstack

Installs an ELK stack
Project URLRSS FeedReport issues

9,721 downloads

9,345 latest version

4.8 quality score

We run a couple of automated
scans to help you access a
module's quality. Each module is
given a score based on how well
the author has formatted their
code and documentation and
modules are also checked for
malware using VirusTotal.

Please note, the information below
is for guidance only and neither of
these methods should be considered
an endorsement by Puppet.

Version information

  • 1.0.1 (latest)
  • 1.0.0
released Apr 14th 2016
This version is compatible with:
  • Puppet Enterprise >= 4.0.0
  • Puppet >= 4.0.0

Start using this module

  • r10k or Code Manager
  • Bolt
  • Manual installation
  • Direct download

Add this module to your Puppetfile:

mod 'loomsen-elkstack', '1.0.1'
Learn more about managing modules with a Puppetfile

Add this module to your Bolt project:

bolt module add loomsen-elkstack
Learn more about using this module with an existing project

Manually install this module globally with Puppet module tool:

puppet module install loomsen-elkstack --version 1.0.1

Direct download is not typically how you would use a Puppet module to manage your infrastructure, but you may want to download the module in order to inspect the code.

Download
Tags: elasticsearch, logstash, kibana, elk

Documentation

loomsen/elkstack — version 1.0.1 Apr 14th 2016

elkstack

Table of Contents

  1. Overview
  2. Module Description - What the module does and why it is useful
  3. Setup - The basics of getting started with elkstack
  4. Configuration - options and additional functionality
  5. Reference - An under-the-hood peek at what the module is doing and how
  6. Limitations - OS compatibility, etc.
  7. Development - Guide for contributing to the module
  8. Editors
  9. Contributors

Overview

The elkstack module installs, configures and manages elkstack on CentOS systems.

Module Description

The elkstack module installs, configures and manages elkstack (elasticsearch, kibana, logstash) For how to configure this, please see below

Notes

This module is best used with an ENC like hiera. It will make your config much easier to read and to maintain. Check the examples to see what I mean.

Setup

What elkstack affects

  • This module will install the elkstack (elasticsearch, kibana, logstash) on your system
  • This module will manage basic configs of each component on your system
  • This module will manage the services on your system

Setup Requirements

Getting Started

You can simply include the class in your config, it should suffice. The defaults should be fine in hiera:

classes:
  - elkstack

Configuration

Here are some more elaborate examples of what you can do with this module.

Examples

This installs the elkstack, install the plugins for the stack, and downloads a PSQL JDBC connector to /usr/share/elasticsearch/lib. It adds a line to kibanas config reading: server.host: localhost

classes:
  - elkstack

elkstack::plugins:
  elasticsearch:
    - 'license'
    - 'marvel-agent'
  logstash:
    - 'logstash-input-jdbc'
  kibana:
    - 'elasticsearch/marvel/latest'
    - 'elastic/sense'
  drivers:
    - 'https://jdbc.postgresql.org/download/postgresql-9.4.1208.jar'

elkstack::with_nginx: false

elkstack::kibana_config:
  - "server.host: localhost"

More options

The defaults are pretty reasonable, I hope. However, you may override pretty much anything. Available parameters are discussed below.

Reference

Classes

Public Classes

  • elkstack: Main class, includes all other classes.

Private Classes

  • elkstack::install: Handles the packages.
  • elkstack::config: Handles configuration files.
  • elkstack::params: default values.
  • elkstack::plugins: Handles the plugins.

Functions

Parameters

The following parameters are available in the ::elkstack class:

$es_config

Elasticsearch configuration. This goes into /etc/elasticsearch/elasticsearch.yml line by line.

Default is: []

hiera example:

elkstack::es_config:
  - 'cluster.name: my-cluster'
  - 'node.name: node1'

$es_main_version

Version for the CentOS repo. You probably want to leave this untouched.

Default is: '2.x'

$kibana_config

Kibana configuration. This goes into /opt/kibana/config/kibana.yml line by line.

Default is: 

  $kibana_config          = [
    'elasticsearch.url: "http://localhost:9200"',
    'server.host: localhost',
  ]

hiera example:

elkstack::kibana_config:
  - 'elasticsearch.url: "http://example.com:9200"'
  - 'server.host: example.com'

$kibana_main_version

Kibana main version to install. Also the version used for creating the CentOS repo.

Default is: '4.5'

$logstash_config_input

Logstash input configuration. This goes into /etc/logstash/conf.d/$filename-input.conf line by line. The hash is of the form: (Please see examples below)

$logstash_config_input = {
  01-filename => {
    input => { 
      "input_configuration=> line1"
      "another_input_config=> line2"
      }
    filter => {
      "filter {"
        "filter line here"
      "}"
    }
    output => {
      "outputname => indexname"
    }
  }
}

Default is:

  $logstash_config_input  = {}

hiera example:

elkstack::logstash_config_input:
  04-psql:
    input:
      - 'jdbc_connection_string => "jdbc:postgresql://example.com:5432/mydb"'
      - 'jdbc_user => "myuser"'
      - 'jdbc_password => "mypassword"'
      - 'jdbc_paging_enabled => "true"'
      - 'jdbc_page_size => "50000"'
      - 'jdbc_validate_connection => true'
      - 'jdbc_driver_library => "/usr/share/elasticsearch/lib/postgresql-9.4.1208.jar"'
      - 'jdbc_driver_class => "org.postgresql.Driver"'
      - 'statement => "SELECT * FROM customer_list"'
    filter:
      - '# just an example'
    output:
      elasticsearch: 'myindex'

Resulting file: /etc/logstash/conf.d/04-psql-input.conf

input {
  psql {
    jdbc_connection_string => "jdbc:postgresql://example.com:5432/mydb"
    jdbc_user => "myuser"
    jdbc_password => "mypassword"
    jdbc_paging_enabled => "true"
    jdbc_page_size => "50000"
    jdbc_validate_connection => true
    jdbc_driver_library => "/usr/share/elasticsearch/lib/postgresql-9.4.1208.jar"
    jdbc_driver_class => "org.postgresql.Driver"
    statement => "SELECT * FROM customer_list"
  }
}
filter {
  # just an example
}
output {
  elasticsearch {
    index => "myindex"
  }
}

$logstash_config_output

Default is:

  $logstash_config_output = {
    '99-elasticsearch' => [
      'hosts           => ["localhost:9200"]',
      'sniffing        => true',
      'manage_template => false',
    ],
  }

Resulting file: /etc/logstash/conf.d/99-elasticsearch-output.conf

output {
  elasticsearch {
    hosts           => ["localhost:9200"]
    sniffing        => true
    manage_template => false
  }
}

$logstash_main_version

logstash main version to install. Also the version used for creating the CentOS repo.

Default is: '2.2'

$package_name

Default packages to install.

Default is: [ 'elasticsearch', 'logstash', 'kibana', 'java' ]

$plugins

Optional plugins to install for each of the stack components. Also you can download additional files, like jdbc drivers here.

Default is:

  $plugins                = {
    elasticsearch => ['license', 'marvel-agent'],
    logstash      => ['logstash-input-jdbc'],
    kibana        => ['elasticsearch/marvel/latest', 'elastic/sense'],
    drivers       => [],
  }

hiera example:

elkstack::plugins:
  elasticsearch:
    - 'license'
    - 'marvel-agent'
  logstash:
    - 'logstash-input-jdbc'
  kibana:
    - 'elasticsearch/marvel/latest'
    - 'elastic/sense'
  drivers:
    - 'https://jdbc.postgresql.org/download/postgresql-9.4.1208.jar'

$service_name

Default services to manage.

Default is: [ 'elasticsearch', 'kibana', 'logstash' ]

$with_nginx

If this is true, nginx will be installed, the service will be managed, and a simple kibana config will be placed in /etc/nginx/conf.d pointing to the systems FQDN.

Default is: true

Limitations

Currently, this module supports CentOS.

Development

I have limited access to resources and time, so if you think this module is useful, like it, hate it, want to make it better or want it off the face of the planet, feel free to get in touch with me.

Editors

Norbert Varzariu (loomsen)

Contributors

Hendrik Horeis hendrik.horeis@gmail.com Additional contributors: please see the list of contributors.