elastic/logstash

A Puppet module for managing and configuring Logstash.

Build Status

Logstash Versions

This module, "elastic/logstash" supports only Logstash 5.x. For earlier Logstash versions, support is provided by the legacy module, "elasticsearch/logstash".

Requirements

  • Puppet 3.8.6 or better.
  • The stdlib Puppet library.

Optional:

  • The apt (>= 2.0.0) Puppet library when using repo management on Debian/Ubuntu.
  • The zypprepo Puppet library when using repo management on SLES/SuSE

Quick Start

This minimum viable configuration ensures that the service is running will be started at boot time.

include logstash

# You must provide a valid pipeline configuration for the service to start.
logstash::configfile { 'my_ls_config':
  content => template('path/to/config.file'),
}

Package and service options

Choosing a Logstash minor version

class { 'logstash':
    version => '5.0.2',
}

Manual repository management

You may want to manage repositories manually. You can disable automatic repository management like this:

class { 'logstash':
  manage_repo  => false,
}

Using an explicit package source

Rather than use your distribution's repository system, you can specify an explicit package to fetch and install.

From an HTTP/HTTPS/FTP URL

class { 'logstash':
  package_url => 'https://artifacts.elastic.co/downloads/logstash/logstash-5.1.1.rpm',
}

From a 'puppet://' URL

class { 'logstash':
  package_url => 'puppet:///modules/my_module/logstash-5.1.1.rpm',
}

From a local file on the agent

class { 'logstash':
  package_url => 'file:///tmp/logstash-5.1.1.rpm',
}

Allow automatic point-release upgrades

class { 'logstash':
  auto_upgrade  => true,
}

Do not run as a service

class { 'logstash':
  status => 'disabled',
}

Disable automatic restarts

Under normal circumstances, changing a configuration will trigger a restart of the service. This behaviour can be disabled:

class { 'logstash':
  restart_on_change => false,
}

Disable and remove Logstash

class { 'logstash':
  ensure => 'absent',
}

Logstash config files

Settings

Logstash uses several files to define settings for the service and associated Java runtime. The settings files can be configured with class parameters.

logstash.yml

class { 'logstash':
  settings => {
    'pipeline.batch.size'  => 25,
    'pipeline.batch.delay' => 5,
  },
}

jvm.options

class { 'logstash':
  jvm_options => [
    '-Xms1g',
    '-Xmx1g',
  ]
}

startup.options

class { 'logstash':
  startup_options => {
    'LS_NICE' => '10',
  }
}

Pipeline Configuration

Pipeline configuration files can be declared with the logstash::configfile type.

logstash::configfile { 'inputs':
  content => template('path/to/input.conf.erb'),
}

or

logstash::configfile { 'filters':
  source => 'puppet:///path/to/filter.conf',
}

For simple cases, it's possible to provide your Logstash config as an inline string:

logstash::configfile { 'basic_ls_config':
  content => 'input { heartbeat {} } output { null {} }',
}

If you want to use hiera to specify your configs, include the following create_resources call in your node manifest or in manifests/site.pp:

$logstash_configs = hiera('logstash_configs', {})
create_resources('logstash::configfile', $logstash_configs)

...and then include the following config within the corresponding hiera file:

"logstash_configs": {
  "config-name": {
    "template": "logstash/config.file.erb",
  }
}

Patterns

Many plugins (notably Grok) use patterns. While many are included in Logstash already, additional site-specific patterns can be managed as well; where possible, you are encouraged to contribute new patterns back to the community.

logstash::patternfile { 'extra_patterns':
  source => 'puppet:///path/to/extra_pattern',
}

By default the resulting filename of the pattern will match that of the source. This can be over-ridden:

logstash::patternfile { 'extra_patterns_firewall':
  source   => 'puppet:///path/to/extra_patterns_firewall_v1',
  filename => 'extra_patterns_firewall',
}

IMPORTANT NOTE: Using logstash::patternfile places new patterns in the correct directory, however, it does NOT cause the path to be included automatically for filters (example: grok filter). You will still need to include this path (by default, /etc/logstash/patterns/) explicitly in your configurations.

Example: If using 'grok' in one of your configurations, you must include the pattern path in each filter like this:

# Note: this example is Logstash configuration, not a Puppet resource.
# Logstash and Puppet look very similar!
grok {
  patterns_dir => "/etc/logstash/patterns/"
  ...
}

Plugin management

Installing by name (from RubyGems.org)

logstash::plugin { 'logstash-input-beats': }

Installing from a local Gem

logstash::plugin { 'logstash-input-custom':
  source => '/tmp/logstash-input-custom-0.1.0.gem',
}

Installing from a 'puppet://' URL

logstash::plugin { 'logstash-filter-custom':
  source => 'puppet:///modules/my_ls_module/logstash-filter-custom-0.1.0.gem',
}

Support

Need help? Join us in #logstash on Freenode IRC or on the https://discuss.elastic.co/c/logstash discussion forum.