Version information
This version is compatible with:
- Puppet Enterprise 2018.1.x, 2017.3.x, 2017.2.x, 2017.1.x, 2016.5.x, 2016.4.x
- Puppet >=4.6.1 <6.0.0
- , , , , , ,
This module has been deprecated by its author since Jan 5th 2023.
The author has suggested puppet-logstash as its replacement.
Start using this module
Documentation
elastic/logstash
A Puppet module for managing and configuring Logstash.
Logstash Versions
This module, "elastic/logstash" supports only Logstash 5.x and 6.x. For earlier Logstash versions, support is provided by the legacy module "elasticsearch/logstash".
Requirements
- Puppet 4.6.1 or better.
- The stdlib module.
- Logstash itself requires Java 8. The "puppetlabs/java" module is recommended for installing Java. This module will not install Java.
Optional:
- The elastic_stack module when using automatic repository management.
- The apt (>= 2.0.0) module when using repo management on Debian/Ubuntu.
- The zypprepo module when using repo management on SLES/SuSE.
Quick Start
This minimum viable configuration ensures that Logstash is installed, enabled, and running:
include logstash
# You must provide a valid pipeline configuration for the service to start.
logstash::configfile { 'my_ls_config':
content => template('path/to/config.file'),
}
Package and service options
Choosing a Logstash minor version
class { 'logstash':
version => '6.0.0',
}
Choosing a Logstash major version
This module uses the related "elastic/elastic_stack" module to manage package repositories. Since there is a separate repository for each major version of the Elastic stack, if you don't want the default version (6), it's necessary to select which version to configure, like this:
class { 'elastic_stack::repo':
version => 5,
}
class { 'logstash':
version => '5.6.4',
}
Manual repository management
You may want to manage repositories manually. You can disable automatic repository management like this:
class { 'logstash':
manage_repo => false,
}
Using an explicit package source
Rather than use your distribution's repository system, you can specify an explicit package to fetch and install.
From an HTTP/HTTPS/FTP URL
class { 'logstash':
package_url => 'https://artifacts.elastic.co/downloads/logstash/logstash-5.1.1.rpm',
}
From a 'puppet://' URL
class { 'logstash':
package_url => 'puppet:///modules/my_module/logstash-5.1.1.rpm',
}
From a local file on the agent
class { 'logstash':
package_url => 'file:///tmp/logstash-5.1.1.rpm',
}
Allow automatic point-release upgrades
class { 'logstash':
auto_upgrade => true,
}
Do not run as a service
class { 'logstash':
status => 'disabled',
}
Disable automatic restarts
Under normal circumstances, changing a configuration will trigger a restart of the service. This behaviour can be disabled:
class { 'logstash':
restart_on_change => false,
}
Disable and remove Logstash
class { 'logstash':
ensure => 'absent',
}
Logstash config files
Settings
Logstash uses several files to define settings for the service and associated Java runtime. The settings files can be configured with class parameters.
logstash.yml
with flat keys
class { 'logstash':
settings => {
'pipeline.batch.size' => 25,
'pipeline.batch.delay' => 5,
}
}
logstash.yml
with nested keys
class { 'logstash':
settings => {
'pipeline' => {
'batch' => {
'size' => 25,
'delay' => 5,
}
}
}
}
jvm.options
class { 'logstash':
jvm_options => [
'-Xms1g',
'-Xmx1g',
]
}
startup.options
class { 'logstash':
startup_options => {
'LS_NICE' => '10',
}
}
pipelines.yml
class { 'logstash':
pipelines => [
{
"pipeline.id" => "pipeline_one",
"path.config" => "/usr/local/etc/logstash/pipeline-1/one.conf",
},
{
"pipeline.id" => "pipeline_two",
"path.config" => "/usr/local/etc/logstash/pipeline-2/two.conf",
}
]
}
Note that specifying pipelines
will automatically remove the default
path.config
setting from logstash.yml
, since this is incompatible with
pipelines.yml
.
Enabling centralized pipeline management with xpack.management.enabled
will
also remove the default path.config
.
Pipeline Configuration
Pipeline configuration files can be declared with the logstash::configfile
type.
logstash::configfile { 'inputs':
content => template('path/to/input.conf.erb'),
}
or
logstash::configfile { 'filters':
source => 'puppet:///path/to/filter.conf',
}
For simple cases, it's possible to provide your Logstash config as an inline string:
logstash::configfile { 'basic_ls_config':
content => 'input { heartbeat {} } output { null {} }',
}
You can also specify the exact path for the config file, which is particularly useful with multiple pipelines:
logstash::configfile { 'config_for_pipeline_two':
content => 'input { heartbeat {} } output { null {} }',
path => '/usr/local/etc/logstash/pipeline-2/two.conf',
}
If you want to use Hiera to specify your configs, include the following create_resources call in your manifest:
create_resources('logstash::configfile', hiera('my_logstash_configs'))
...and then create a data structure like this in Hiera:
---
my_logstash_configs:
nginx:
template: site_logstash/nginx.conf.erb
syslog:
template: site_logstash/syslog.conf.erb
In this example, templates for the config files are stored in the custom,
site-specific module "site_logstash
".
Patterns
Many plugins (notably Grok) use patterns. While many are included in Logstash already, additional site-specific patterns can be managed as well.
logstash::patternfile { 'extra_patterns':
source => 'puppet:///path/to/extra_pattern',
}
By default the resulting filename of the pattern will match that of the source. This can be over-ridden:
logstash::patternfile { 'extra_patterns_firewall':
source => 'puppet:///path/to/extra_patterns_firewall_v1',
filename => 'extra_patterns_firewall',
}
IMPORTANT NOTE: Using logstash::patternfile places new patterns in the correct directory, however, it does NOT cause the path to be included automatically for filters (example: grok filter). You will still need to include this path (by default, /etc/logstash/patterns/) explicitly in your configurations.
Example: If using 'grok' in one of your configurations, you must include the pattern path in each filter like this:
# Note: this example is Logstash configuration, not a Puppet resource.
# Logstash and Puppet look very similar!
grok {
patterns_dir => "/etc/logstash/patterns/"
...
}
Plugin management
Installing by name (from RubyGems.org)
logstash::plugin { 'logstash-input-beats': }
Installing from a local Gem
logstash::plugin { 'logstash-input-custom':
source => '/tmp/logstash-input-custom-0.1.0.gem',
}
Installing from a 'puppet://' URL
logstash::plugin { 'logstash-filter-custom':
source => 'puppet:///modules/my_ls_module/logstash-filter-custom-0.1.0.gem',
}
Installing from an 'http(s)://' URL
logstash::plugin { 'x-pack':
source => 'https://artifacts.elastic.co/downloads/packs/x-pack/x-pack-5.3.0.zip',
}
Controling the environment for the logstash-plugin
command
logstash::plugin { 'logstash-input-websocket':
environment => 'LS_JVM_OPTS="-Xms1g -Xmx1g"',
}
Support
Need help? Join us in #logstash on Freenode IRC or on the https://discuss.elastic.co/c/logstash discussion forum.
6.1.5
Honour restart_on_change
for startup files #377
6.1.4 Use Upstart for init on Oracle Linux 6
6.1.3 Allow puppetlabs-stdlib 5.x
6.1.2 Set cwd to "/tmp" during plugin execs.
6.1.1 Update init system handling.
6.1.0 Support centralized pipeline management.
6.0.1 Fix explicit versions like "5.6.2-1.noarch" #353
6.0.0 Puppet 3 support removed. Minimum Puppet version is now 4.6.1. Puppet 5 supported. Logstash 6.x (and 5.x) supported. File permissions for config files now match those from the Logstash package. elastic/elastic_stack module is now used to manage repositories. Logstash multiple pipelines supported. Config files resources accept an explicit target path.
5.3.0 Allow setting environment for plugin resources
5.2.0 Allow 'http(s):// URLs for plugin install.
5.1.0 Make config files be owned by root. Allow 'file://' URLs for plugin install. Sort lines in jvm.options for file.
5.0.4 Expose $logstash::home_dir
5.0.3 Sort startup_options to prevent triggering unneeded restarts.
5.0.2 Do not autmatically add ".conf" to pipeline config filesnames.
5.0.1 Trivial README update.
5.0.0 Major re-write for Logstash 5.x. Drop support for Logstash <5.0.0.
0.6.4 Puppet 4 support.
0.6.3 Documentation updates only. Functionally identical to 0.6.2.
0.6.2 Allow electrical/file_concat version 1.x.
0.6.1 Restart service on pattern file change. Remove dependency on external PGP server. Fix circular dependency on plugin installation. 0.6.0 Deprecates the logstash-contrib package. Supports Gem-based Logstash plugins. Not compatible with Logstash versions < 1.5.0.
0.5.1 Updated system tests to work with LS 1.4.1 Increase package download timeout Add option to use stages for the repo setup instead anchors
0.5.0 Move beaker testing to use docker Large module update to work with the contrib package Refactored rspec testing Fix inline docs Added Puppet 3.5.0 testing Fix typo in group name variable ( PR #147 ) Improve puppet module removal ( PR #149 ) Reverted PR #149. Caused issues with package removal. added lsbdistid = Debian to rspec facts ( PR #146 ) match other config perms with patterns ( PR #151 )
0.4.3 Lower puppetlabs-stdlib depdency from 4.0.0 to 3.2.0 Documentation improvements ( PR #132 #137 ) Fixed yumrepo call to include description ( PR #138 ) Added beaker testing Fixed bug that sometimes LS starts before all configs are processed. Ensure java is installed before installing the package when using package_url Fail fast when using package_url and repo config
0.4.2 Fix config directory for config files to be inline with the init file of the packages Update readme ( thanks to PR #130 from phrawzty ) Added repo management ( based on work of PR #121 from pcfens )
0.4.1 Important Update Ensure exec names are unique. This caused an issue when using the Elasticsearch Puppet module Removed a part in the package.pp that should have been removed ( missed with the rewrite ) Missed a few bits of the rewrite. Updated readme to reflect reality regarding configfile define.
0.4.0 NOTE: This is a backwards compability breaking release !! Large rewrite of the entire module described below Make the core more dynamic for different service providers Add better testing and devided into different files Add different ways to install the package except from the repository ( puppet/http/https/ftp/file ) Update java class to install openjdk 1.7 Add validation of templates Added more test scenario's Added puppet parser validate task for added checking Improve module removing when set to absent Updated readme Doc improvements by dan ( phrawzty ) Added define for managing pattern files Added define for managing plugins
0.3.4 Fixing purging of created directories ( PR #61, #64 by Kayla Green and Jason Koppe ) Documentation fixes ( PR #65, #67 by Kristian Glass and Andreas Paul ) Making config dir configurable ( PR #70 by Justin Lambert ) Permit HTTP(s) for downloading logstash ( PR #71 by Phil Fenstermacher ) Ensure user/group is passed in the debian init file Spec cleanup ( PR #75 by Justin Lambert ) set logstash logdir perms when using custom jar provider ( PR #74 by Justin Lambert ) clean up installpath when updating jars ( PR #72 by Justin Lambert ) fix wrong creates path at jar custom provider ( PR #83 by Daniel Werdermann ) added 'in progress' for logstash version 1.2.x ( PR #87 by rtoma ) Add small input/output examples ( PR #89 by Andreas Paul ) Solving defaults file not being installed in some cases http download of jar should require $jardir ( PR #90 by Max Griffiths ) add ability to install a logstash config file ( PR #93 by Justin Lambert )
0.3.3 Enable puppet 3.2.x testing Fix issue that the config dir was missing in the init files Fix variable access deprecation warning ( PR #56 by Richard Peng )
0.3.2 Fixing issue when using jar file without multi-instance feature Added rspec tests to cover this issue
0.3.1 Missed changes for enabling/disabling multi-instance feature Adding a few spec tests for the multi-instance feature
0.3.0 Update defines for Logstash 1.1.12 Adding license file Deleted old init file removal to avoid issues. ( Issue #50 ) Allow file owner/group to be variable ( Issue/PR #47 ) Ensure log directory exists before starting ( PR #53 by Brian Lalor ) Provide complete containment of the class ( PR #53 by Brian Lalor ) Update rspec tests for new defines
0.2.0 Update defines for logstash 1.1.10 New feature for plugins to automatically transfer files ( Issue #24 ) Create correct tmp dir ( Issue #35 ) Change file modes to be more secure ( Issue #36 ) Update defines for better input validation ( Issue #43 ) Adding rspec tests for plugin defines Fix tmp dir Debian init script ( PR #44 by Dan Carley )
0.1.0 Don't backup the Jar file or the symlink ( Issue #25 by Garth Kidd ) First implementation of the multi-instance feature. This will break certain functionality.
0.0.6 Fix issue that the init file was overwritten Ensure we install java first before starting logstash if enabled
0.0.5 Adding spec tests Update Readme ( PR #20 by rjw1 ) New feature to install java
0.0.4 Rename Redhat to RedHat for init file ( PR #12 by pkubat ) Adding Amazon as Operating system ( PR #12 by pkubat ) Symlinking Jar file to generic name ( PR #12 by pkubat ) Correting symlink ( PR #14 by Jeff Wong )
0.0.3 Clarify jarfile usage and validation ( PR #6 by Garth Kidd ) Add default Debian Init script when non provided and using custom source ( PR #7 by Garth Kidd ) Add RedHat as OS type ( PR #8 by Dan ) Skip init script when status = unmanaged ( PR #9 by Tavis Aitken ) Refactored the custom provider part ( With help of Garth Kidd )
0.0.2 Adding a way to provide jar and init file instead of depending on a package
0.0.1 Initial release of the module
Dependencies
- puppetlabs/stdlib (>=3.2.0 <6.0.0)
- elastic/elastic_stack (>=6.0.0 <7.0.0)
Copyright (c) 2012-2016 Elasticsearch <http://www.elastic.co> Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.