logstash

A Puppet module for managing and configuring Logstash.

Version 7 and newer of this module are released by Vox Pupuli. They now follow semantic versioning. Previously the module was maintained by Elastic.

Requirements

• The stdlib module.
• Logstash < 7.0.0 requires Java. The puppetlabs/java module is recommended for installing Java.

Optional:

• The elastic_stack module when using automatic repository management.
• The apt (>= 2.0.0) module when using repo management on Debian/Ubuntu.
• The zypprepo module when using repo management on SLES/SuSE.

Quick Start

This minimum viable configuration ensures that Logstash is installed, enabled, and running:

include logstash

# You must provide a valid pipeline configuration for the service to start.
logstash::configfile { 'my_ls_config':
  content => template('path/to/config.file'),
}

Package and service options

Choosing a Logstash major version

This module uses the related "elastic/elastic_stack" module to manage package repositories. Since there is a separate repository for each major version of the Elastic stack, if you don't want the default version, it's necessary to select which version to configure, like this:

class { 'elastic_stack::repo':
  version => 6,
}

include logstash

Choosing a Logstash minor version

class { 'logstash':
  version => '6.8.0',
}

Manual repository management

You may want to manage repositories manually. You can disable automatic repository management like this:

class { 'logstash':
  manage_repo => false,
}

Using an explicit package source

Rather than use your distribution's repository system, you can specify an explicit package to fetch and install.

From an HTTP/HTTPS/FTP URL

class { 'logstash':
  package_url => 'https://artifacts.elastic.co/downloads/logstash/logstash-7.17.8-x86_64.rpm',
}

From a 'puppet://' URL

class { 'logstash':
  package_url => 'puppet:///modules/my_module/logstash-7.17.8-x86_64.rpm',
}

From a local file on the agent

class { 'logstash':
  package_url => 'file:///tmp/logstash-7.17.8-x86_64.rpm',
}

Allow automatic point-release upgrades

class { 'logstash':
  auto_upgrade => true,
}

Use a different logstash home

class { 'logstash':
  home_dir => '/opt/logstash',
}

Do not run as a service

class { 'logstash':
  status => 'disabled',
}

Disable automatic restarts

Under normal circumstances, changing a configuration will trigger a restart of the service. This behaviour can be disabled:

class { 'logstash':
  restart_on_change => false,
}

Disable and remove Logstash

class { 'logstash':
  ensure => 'absent',
}

Logstash config files

Settings

Logstash uses several files to define settings for the service and associated Java runtime. The settings files can be configured with class parameters.

logstash.yml with flat keys

class { 'logstash':
  settings => {
    'pipeline.batch.size'  => 25,
    'pipeline.batch.delay' => 5,
  }
}

logstash.yml with nested keys

class { 'logstash':
  settings => {
    'pipeline' => {
      'batch' => {
        'size'  => 25,
        'delay' => 5,
      }
    }
  }
}

jvm.options

class { 'logstash':
  jvm_options => [
    '-Xms1g',
    '-Xmx1g',
  ]
}

startup.options

class { 'logstash':
  startup_options => {
    'LS_NICE' => '10',
  }
}

pipelines.yml

class { 'logstash':
  pipelines => [
    {
      "pipeline.id" => "pipeline_one",
      "path.config" =>  "/usr/local/etc/logstash/pipeline-1/one.conf",
    },
    {
      "pipeline.id" => "pipeline_two",
      "path.config" =>  "/usr/local/etc/logstash/pipeline-2/two.conf",
    }
  ]
}

Note that specifying pipelines will automatically remove the default path.config setting from logstash.yml, since this is incompatible with pipelines.yml.

Enabling centralized pipeline management with xpack.management.enabled will also remove the default path.config.

Pipeline Configuration

Pipeline configuration files can be declared with the logstash::configfile type.

logstash::configfile { 'inputs':
  content => template('path/to/input.conf.erb'),
}

or

logstash::configfile { 'filters':
  source => 'puppet:///path/to/filter.conf',
}

For simple cases, it's possible to provide your Logstash config as an inline string:

logstash::configfile { 'basic_ls_config':
  content => 'input { heartbeat {} } output { null {} }',
}

You can also specify the exact path for the config file, which is particularly useful with multiple pipelines:

logstash::configfile { 'config_for_pipeline_two':
  content => 'input { heartbeat {} } output { null {} }',
  path    => '/usr/local/etc/logstash/pipeline-2/two.conf',
}

If you want to use Hiera to specify your configs, include the following create_resources call in your manifest:

create_resources('logstash::configfile', hiera('my_logstash_configs'))

...and then create a data structure like this in Hiera:

---
my_logstash_configs:
  nginx:
    template: site_logstash/nginx.conf.erb
  syslog:
    template: site_logstash/syslog.conf.erb

In this example, templates for the config files are stored in the custom, site-specific module "site_logstash".

Patterns

Many plugins (notably Grok) use patterns. While many are included in Logstash already, additional site-specific patterns can be managed as well.

logstash::patternfile { 'extra_patterns':
  source => 'puppet:///path/to/extra_pattern',
}

By default the resulting filename of the pattern will match that of the source. This can be over-ridden:

logstash::patternfile { 'extra_patterns_firewall':
  source   => 'puppet:///path/to/extra_patterns_firewall_v1',
  filename => 'extra_patterns_firewall',
}

IMPORTANT NOTE: Using logstash::patternfile places new patterns in the correct directory, however, it does NOT cause the path to be included automatically for filters (example: grok filter). You will still need to include this path (by default, /etc/logstash/patterns/) explicitly in your configurations.

Example: If using 'grok' in one of your configurations, you must include the pattern path in each filter like this:

# Note: this example is Logstash configuration, not a Puppet resource.
# Logstash and Puppet look very similar!
grok {
  patterns_dir => "/etc/logstash/patterns/"
  ...
}

Plugin management

Installing by name (from RubyGems.org)

logstash::plugin { 'logstash-input-beats': }

Installing from a local Gem

logstash::plugin { 'logstash-input-custom':
  source => '/tmp/logstash-input-custom-0.1.0.gem',
}

Installing from a 'puppet://' URL

logstash::plugin { 'logstash-filter-custom':
  source => 'puppet:///modules/my_ls_module/logstash-filter-custom-0.1.0.gem',
}

Installing from an 'http(s)://' URL

logstash::plugin { 'x-pack':
  source => 'https://artifacts.elastic.co/downloads/packs/x-pack/x-pack-5.3.0.zip',
}

Controling the environment for the logstash-plugin command

logstash::plugin { 'logstash-input-websocket':
  environment => 'LS_JAVA_OPTS=-Xms1g -Xmx1g',
}

Transfer Notice

This module was originally authored by Elastic. The maintainer preferred that Vox Pupuli take ownership of the module for future improvement and maintenance. Existing pull requests and issues were transferred over, please fork and continue to contribute here instead of Elastic.

Reference

Table of Contents

Classes

• logstash: This class manages installation, configuration and execution of Logstash 5.x.
• logstash::config: This class manages configuration directories for Logstash.
• logstash::package: This class manages the Logstash package. It is usually used only by the top-level logstash class. It's unlikely that you will need to decl
• logstash::service: This mangages the system service for Logstash. It is usually used only by the top-level logstash class. It's unlikely that you will need t

Defined types

• logstash::configfile: This type represents a Logstash pipeline configuration file. Parameters are mutually exclusive. Only one should be specified.
• logstash::patternfile: This type represents a Grok pattern file for Logstash.
• logstash::plugin: Manage the installation of a Logstash plugin. By default, plugins are downloaded from RubyGems, but it is also possible to install from a lo

Classes

logstash

This class manages installation, configuration and execution of Logstash 5.x.

Examples

Install Logstash, ensure the service is running and enabled.

class { 'logstash': }

Remove Logstash.

class { 'logstash':
  ensure => 'absent',
}

Install everything but disable the service.

class { 'logstash':
  status => 'disabled',
}

Configure Logstash settings.

class { 'logstash':
  settings => {
    'http.port' => '9700',
  }
}

Configure Logstash startup options.

class { 'logstash':
  startup_options => {
    'LS_USER' => 'root',
  }
}

Set JVM memory options.

class { 'logstash':
  jvm_options => [
    '-Xms1g',
    '-Xmx1g',
  ]
}

Configure multiple pipelines.

class { 'logstash':
  pipelines => [
    {
      "pipeline.id" => "my-pipeline_1",
      "path.config" =>  "/etc/path/to/p1.config",
    },
    {
      "pipeline.id" => "my-other-pipeline",
      "path.config" =>  "/etc/different/path/p2.cfg",
    }
  ]
}

Parameters

The following parameters are available in the logstash class:

• ensure
• auto_upgrade
• status
• version
• restart_on_change
• package_url
• package_name
• download_timeout
• home_dir
• logstash_user
• logstash_group
• purge_config
• service_provider
• settings
• startup_options
• jvm_options_defaults
• jvm_options
• pipelines
• manage_repo
• config_dir

ensure

Data type: String

Controls if Logstash should be present or absent.

If set to absent, the Logstash package will be uninstalled. Related files will be purged as much as possible. The exact behavior is dependant on the service provider, specifically its support for the 'purgable' property.

Default value: 'present'

auto_upgrade

Data type: Boolean

If set to true, Logstash will be upgraded if the package provider is able to find a newer version. The exact behavior is dependant on the service provider, specifically its support for the 'upgradeable' property.

Default value: false

status

Data type: String

The desired state of the Logstash service. Possible values:

• enabled: Service running and started at boot time.
• disabled: Service stopped and not started at boot time.
• running: Service running but not be started at boot time.
• unmanaged: Service will not be started at boot time. Puppet will neither stop nor start the service.

Default value: 'enabled'

version

Data type: String

The specific version to install, if desired.

Default value: undef

restart_on_change

Data type: Boolean

Restart the service whenever the configuration changes.

Disabling automatic restarts on config changes may be desired in an environment where you need to ensure restarts occur in a controlled/rolling manner rather than during a Puppet run.

Default value: true

package_url

Data type: String

Explict Logstash package URL to download.

Valid URL types are:

• http://
• https://
• ftp://
• puppet://
• file:/

Default value: undef

package_name

Data type: String

The name of the Logstash package in the package manager.

Default value: 'logstash'

download_timeout

Data type: Integer

Timeout, in seconds, for http, https, and ftp downloads.

Default value: 600

home_dir

Data type: Stdlib::Absolutepath

The home directory for logstash.

Default value: '/usr/share/logstash'

logstash_user

Data type: String

The user that Logstash should run as. This also controls file ownership.

Default value: 'logstash'

logstash_group

Data type: String

The group that Logstash should run as. This also controls file group ownership.

Default value: 'logstash'

purge_config

Data type: Boolean

Purge the config directory of any unmanaged files,

Default value: true

service_provider

Data type: Optional[String[1]]

Service provider (init system) to use. By Default, the module will try to choose the 'standard' provider for the current distribution.

Default value: undef

settings

Data type: Hash

A collection of settings to be defined in logstash.yml.

See: https://www.elastic.co/guide/en/logstash/current/logstash-settings-file.html

Default value: {}

startup_options

Data type: Hash

A collection of settings to be defined in startup.options.

See: https://www.elastic.co/guide/en/logstash/current/config-setting-files.html

Default value: {}

jvm_options_defaults

Data type: Hash

Default set of optionname => option mappings from upstream 8.5 version

Default value:

{
    '-Xms' => '-Xms1g',
    '-Xmx' => '-Xmx1g',
    'UseConcMarkSweepGC' => '11-13:-XX:+UseConcMarkSweepGC',
    'CMSInitiatingOccupancyFraction=' => '11-13:-XX:CMSInitiatingOccupancyFraction=75',
    'UseCMSInitiatingOccupancyOnly' => '11-13:-XX:+UseCMSInitiatingOccupancyOnly',
    '-Djava.awt.headless=' => '-Djava.awt.headless=true',
    '-Dfile.encoding=' => '-Dfile.encoding=UTF-8',
    'HeapDumpOnOutOfMemoryError' => '-XX:+HeapDumpOnOutOfMemoryError',
    '-Djava.security.egd' => '-Djava.security.egd=file:/dev/urandom',
  }

jvm_options

Data type: Array

A collection of settings to be defined in jvm.options. Override same settings in jvm_options_defaults

Default value: []

pipelines

Data type: Array

A collection of settings to be defined in pipelines.yml.

Default value: []

manage_repo

Data type: Boolean

Enable repository management. Configure the official repositories.

Default value: true

config_dir

Data type: String

Path containing the Logstash configuration.

Default value: '/etc/logstash'

logstash::config

This class manages configuration directories for Logstash.

Examples

Include this class to ensure its resources are available.

include logstash::config

logstash::package

This class manages the Logstash package.

It is usually used only by the top-level logstash class. It's unlikely that you will need to declare this class yourself.

Examples

Include this class to ensure its resources are available.

include logstash::package

Parameters

The following parameters are available in the logstash::package class:

• package_name
• version
• package_url

package_name

Data type: String

The name of the Logstash package in the package manager.

Default value: $logstash::package_name

version

Data type: String

Install precisely this version from the package manager.

Default value: $logstash::version

package_url

Data type: String

Get the package from this URL, not from the package manager.

Default value: $logstash::package_url

logstash::service

This mangages the system service for Logstash.

It is usually used only by the top-level logstash class. It's unlikely that you will need to declare this class yourself.

Examples

Include this class to ensure its resources are available.

include logstash::service

Defined types

logstash::configfile

This type represents a Logstash pipeline configuration file.

Parameters are mutually exclusive. Only one should be specified.

Examples

Create a config file content with literal content.

logstash::configfile { 'heartbeat':
  content => 'input { heartbeat {} }',
}

Render a config file from a template.

logstash::configfile { 'from-template':
  template => 'site-logstash-module/pipeline-config.erb',
}

Copy the config from a file source.

logstash::configfile { 'apache':
  source => 'puppet://path/to/apache.conf',
}

Create a config at specific location. Good for multiple pipelines.

logstash::configfile { 'heartbeat-2':
  content => 'input { heartbeat {} }',
  path    => '/usr/local/etc/logstash/pipeline-2/heartbeat.conf'
}

Parameters

The following parameters are available in the logstash::configfile defined type:

• content
• template
• source
• path

content

Data type: String

Literal content to be placed in the file.

Default value: undef

template

Data type: String

A template from which to render the file.

Default value: undef

source

Data type: String

A file resource to be used for the file.

Default value: undef

path

Data type: String

An optional full path at which to create the file.

Default value: undef

logstash::patternfile

This type represents a Grok pattern file for Logstash.

Examples

Define a pattern file.

logstash::patternfile { 'mypattern':
  source => 'puppet:///path/to/my/custom/pattern'
}

Define a pattern file with an explicit destination filename.

logstash::patternfile { 'mypattern':
  source   => 'puppet:///path/to/my/custom/pattern',
  filename => 'custom-pattern-name'
}

Parameters

The following parameters are available in the logstash::patternfile defined type:

• source
• filename

source

Data type: Pattern[/^(puppet|file):\/\//]

File source for the pattern file. eg. puppet://[...] or file://[...]

Default value: undef

filename

Data type: Optional[String[1]]

Optionally set the destination filename.

Default value: undef

logstash::plugin

Manage the installation of a Logstash plugin.

By default, plugins are downloaded from RubyGems, but it is also possible to install from a local Gem, or one stored in Puppet.

Examples

Install a plugin.

logstash::plugin { 'logstash-input-stdin': }

Remove a plugin.

logstash::plugin { 'logstash-input-stout':
  ensure => absent,
}

Install a plugin from a local file.

logstash::plugin { 'logstash-input-custom':
  source => 'file:///tmp/logstash-input-custom.gem',
}

Install a plugin from a Puppet module.

logstash::plugin { 'logstash-input-custom':
  source => 'puppet:///modules/logstash-site-plugins/logstash-input-custom.gem',
}

Install X-Pack.

logstash::plugin { 'x-pack':
  source => 'https://artifacts.elastic.co/downloads/packs/x-pack/x-pack-5.3.0.zip',
}

Install a plugin, overriding JVM options via the environment.

logstash::plugin { 'logstash-input-jmx':
  environment => ['LS_JAVA_OPTS=-Xms1g -Xmx1g']
}

Parameters

The following parameters are available in the logstash::plugin defined type:

• ensure
• source
• user
• environment

ensure

Data type: String

Install or remove with present or absent.

Default value: present

source

Data type: String

Install from this file, not from RubyGems.

Default value: undef

user

Data type: String

User to install plugin as.

Default value: 'root'

environment

Data type: String

Environment used when running 'logstash-plugin'

Default value: []

Changelog

All notable changes to this project will be documented in this file. Each new release typically also includes the latest modulesync defaults. These should not affect the functionality of the module.

v8.1.0 (2024-03-26)

Full Changelog

Implemented enhancements:

• Add Rocky/AlmaLinux/OracleLinux support #442 (bastelfreak)
• Add Ubuntu 22.04 support #441 (bastelfreak)
• Add EL9 support #440 (bastelfreak)
• Add Debian 12 support #439 (bastelfreak)
• Don't hardcode the service provider #438 (bastelfreak)
• replace legacy merge() with native puppet code #437 (bastelfreak)

v8.0.0 (2024-01-30)

Full Changelog

Breaking changes:

• Drop Puppet 6 support #424 (bastelfreak)

Implemented enhancements:

• bump elastic_stack max version to \<10.0.0 #432 (sandwitch)
• Add Puppet 8 support #427 (bastelfreak)
• puppetlabs/stdlib: Allow 9.x #426 (bastelfreak)

v7.0.0 (2023-01-05)

Full Changelog

Breaking changes:

• Fix option spelling in jvm.options template #412 (travees)

Implemented enhancements:

• allow overriding default jvm options #417 (juliantaylor)
• Add a parameter to override the plugin install user #414 (thebeanogamer)
• New parameter for home directory #368 (joernott)

Closed issues:

• Cannot override JVM options for plugins #381

Merged pull requests:

• Finalize voxpupuli transition #420 (h-haaks)
• metadata.json: Adjust modulename/GitHub URLs #416 (bastelfreak)
• migrate .fixtures.yml to git repos #410 (bastelfreak)
• puppet-lint: autofix #409 (bastelfreak)
• Stop using Travis CI #406 (jmlrt)
• bump up dependencies #405 (anesterova)

6.1.5 (2018-11-13)

Honour restart_on_change for startup files #377

6.1.4 Use Upstart for init on Oracle Linux 6

6.1.3 Allow puppetlabs-stdlib 5.x

6.1.2 Set cwd to "/tmp" during plugin execs.

6.1.1 Update init system handling.

6.1.0 Support centralized pipeline management.

6.0.1 Fix explicit versions like "5.6.2-1.noarch" #353

6.0.0 Puppet 3 support removed. Minimum Puppet version is now 4.6.1. Puppet 5 supported. Logstash 6.x (and 5.x) supported. File permissions for config files now match those from the Logstash package. elastic/elastic_stack module is now used to manage repositories. Logstash multiple pipelines supported. Config files resources accept an explicit target path.

5.3.0 Allow setting environment for plugin resources

5.2.0 Allow 'http(s):// URLs for plugin install.

5.1.0 Make config files be owned by root. Allow 'file://' URLs for plugin install. Sort lines in jvm.options for file.

5.0.4 Expose $logstash::home_dir

5.0.3 Sort startup_options to prevent triggering unneeded restarts.

5.0.2 Do not autmatically add ".conf" to pipeline config filesnames.

5.0.1 Trivial README update.

5.0.0 Major re-write for Logstash 5.x. Drop support for Logstash <5.0.0.

0.6.4 Puppet 4 support.

0.6.3 Documentation updates only. Functionally identical to 0.6.2.

0.6.2 Allow electrical/file_concat version 1.x.

0.6.1 Restart service on pattern file change. Remove dependency on external PGP server. Fix circular dependency on plugin installation. 0.6.0 Deprecates the logstash-contrib package. Supports Gem-based Logstash plugins. Not compatible with Logstash versions < 1.5.0.

0.5.1 Updated system tests to work with LS 1.4.1 Increase package download timeout Add option to use stages for the repo setup instead anchors

0.5.0 Move beaker testing to use docker Large module update to work with the contrib package Refactored rspec testing Fix inline docs Added Puppet 3.5.0 testing Fix typo in group name variable ( PR #147 ) Improve puppet module removal ( PR #149 ) Reverted PR #149. Caused issues with package removal. added lsbdistid = Debian to rspec facts ( PR #146 ) match other config perms with patterns ( PR #151 )

0.4.3 Lower puppetlabs-stdlib depdency from 4.0.0 to 3.2.0 Documentation improvements ( PR #132 #137 ) Fixed yumrepo call to include description ( PR #138 ) Added beaker testing Fixed bug that sometimes LS starts before all configs are processed. Ensure java is installed before installing the package when using package_url Fail fast when using package_url and repo config

0.4.2 Fix config directory for config files to be inline with the init file of the packages Update readme ( thanks to PR #130 from phrawzty ) Added repo management ( based on work of PR #121 from pcfens )

0.4.1 Important Update Ensure exec names are unique. This caused an issue when using the Elasticsearch Puppet module Removed a part in the package.pp that should have been removed ( missed with the rewrite ) Missed a few bits of the rewrite. Updated readme to reflect reality regarding configfile define.

0.4.0 NOTE: This is a backwards compability breaking release !! Large rewrite of the entire module described below Make the core more dynamic for different service providers Add better testing and devided into different files Add different ways to install the package except from the repository ( puppet/http/https/ftp/file ) Update java class to install openjdk 1.7 Add validation of templates Added more test scenario's Added puppet parser validate task for added checking Improve module removing when set to absent Updated readme Doc improvements by dan ( phrawzty ) Added define for managing pattern files Added define for managing plugins

0.3.4 Fixing purging of created directories ( PR #61, #64 by Kayla Green and Jason Koppe ) Documentation fixes ( PR #65, #67 by Kristian Glass and Andreas Paul ) Making config dir configurable ( PR #70 by Justin Lambert ) Permit HTTP(s) for downloading logstash ( PR #71 by Phil Fenstermacher ) Ensure user/group is passed in the debian init file Spec cleanup ( PR #75 by Justin Lambert ) set logstash logdir perms when using custom jar provider ( PR #74 by Justin Lambert ) clean up installpath when updating jars ( PR #72 by Justin Lambert ) fix wrong creates path at jar custom provider ( PR #83 by Daniel Werdermann ) added 'in progress' for logstash version 1.2.x ( PR #87 by rtoma ) Add small input/output examples ( PR #89 by Andreas Paul ) Solving defaults file not being installed in some cases http download of jar should require $jardir ( PR #90 by Max Griffiths ) add ability to install a logstash config file ( PR #93 by Justin Lambert )

0.3.3 Enable puppet 3.2.x testing Fix issue that the config dir was missing in the init files Fix variable access deprecation warning ( PR #56 by Richard Peng )

0.3.2 Fixing issue when using jar file without multi-instance feature Added rspec tests to cover this issue

0.3.1 Missed changes for enabling/disabling multi-instance feature Adding a few spec tests for the multi-instance feature

0.3.0 Update defines for Logstash 1.1.12 Adding license file Deleted old init file removal to avoid issues. ( Issue #50 ) Allow file owner/group to be variable ( Issue/PR #47 ) Ensure log directory exists before starting ( PR #53 by Brian Lalor ) Provide complete containment of the class ( PR #53 by Brian Lalor ) Update rspec tests for new defines

0.2.0 Update defines for logstash 1.1.10 New feature for plugins to automatically transfer files ( Issue #24 ) Create correct tmp dir ( Issue #35 ) Change file modes to be more secure ( Issue #36 ) Update defines for better input validation ( Issue #43 ) Adding rspec tests for plugin defines Fix tmp dir Debian init script ( PR #44 by Dan Carley )

0.1.0 Don't backup the Jar file or the symlink ( Issue #25 by Garth Kidd ) First implementation of the multi-instance feature. This will break certain functionality.

0.0.6 Fix issue that the init file was overwritten Ensure we install java first before starting logstash if enabled

0.0.5 Adding spec tests Update Readme ( PR #20 by rjw1 ) New feature to install java

0.0.4 Rename Redhat to RedHat for init file ( PR #12 by pkubat ) Adding Amazon as Operating system ( PR #12 by pkubat ) Symlinking Jar file to generic name ( PR #12 by pkubat ) Correting symlink ( PR #14 by Jeff Wong )

0.0.3 Clarify jarfile usage and validation ( PR #6 by Garth Kidd ) Add default Debian Init script when non provided and using custom source ( PR #7 by Garth Kidd ) Add RedHat as OS type ( PR #8 by Dan ) Skip init script when status = unmanaged ( PR #9 by Tavis Aitken ) Refactored the custom provider part ( With help of Garth Kidd )

0.0.2 Adding a way to provide jar and init file instead of depending on a package

0.0.1 Initial release of the module

* This Changelog was automatically generated by github_changelog_generator