Filebeats

Table of Contents

1. Description
2. Setup - The basics of getting started with filebeats
   • Setup requirements
   • Beginning with filebeats
3. Usage - Configuration options and additional functionality
4. Reference - An under-the-hood peek at what the module is doing and how
5. Limitations - OS compatibility, etc.
6. Development - Guide for contributing to the module

Description

Very simple puppet module to install and configure elastic search file beats

Setup

puppet module install hetzner-filebeats

Setup Requirements

Puppet labs APT module version 1.8.0 >=

Puppet labs STDLIB module version 4.6.0 >=

Beginning with filebeats

Use puppet module install function to install module and simply include it from your enc/profile/role/site.pp.

Usage

The module can be called with the following parameters:

*export_log_paths

An array of Strings that specifies which logs the filebeats application must export.

*prospectors OPTIONAL

An array of Hashes that specifies which groups of prospectors log entries the filebeats application must export. This value should be used if you wish to have more than one prospector.

*shield_username

The username filebeats should use to authenticate should your cluster make use of shield

*shield_password

The password filebeats should use to authenticate should your cluster make use of shield

*elasticsearch_proxy_host

A string containing the hostname of your proxy host used for load balancing your cluster. If left empty it will default to exporting logs to your local host on port 9200.

*elasticsearch_protocol

A string containing the protocol used by filebeats, defaults to http.

*tls_certificate_authorities

An array of Strings that specifies paths to Certificate authority files.

*tls_certificate

A String that specifies a path to your hosts certificate to use when connecting to elasticsearch.

*tls_certificate_key

A String that specifies a path to your hosts certificate key to use when connecting to elasticsearch.

*log_settings

A puppet Hash containing log level ('debug', 'warning', 'error' or 'critical'), to_syslog(true/false), path('/var/log/filebeat'), keepfiles(7), rotateeverybytes(10485760), name(filebeats.log)

*service_state

A string to describe the state of the filebeats service ('stopped'/'running'), defaults to 'running'

*loadbalance

A boolean to turn on or off load balancing for logstash outputs, defulats to false.

*logstash_hosts

An array of strings that specifies remote hosts to use for logstash outputs, e.g ['localhost:5044']

*logstash_index

A string that specifies the index to use for the logstash output, defaults to '[filebeat-]YYYY.MM.DD' as per the package.

*elasticsearch_index

A string that specifies the index to use for the elasticsearch output, defaults to '[filebeat-]YYYY.MM.DD' as per the package.

Example

Auth.log being exported with only shield login details specified.

   class { 'filebeats':
     export_log_paths         => ['/var/log/auth.log'],
     shield_username          => 'host',
     shield_password          => 'secret',
     elasticsearch_proxy_host => 'elasticsearchproxy.myserver.com',
   }

Multiple prospectors with multiple log files being exported.

   class { 'filebeats':
     prospectors              => [{ 'input_type' => 'log',
                                    'doc_type'   => 'log',
                                    'paths'      => ['/var/log/auth.log']
                                  },
                                  { 'input_type' => 'log',
                                    'doc_type'   => 'apache',
                                    'paths'      => ['/var/log/apache2/access.log', '/var/log/apache2/error.log'],
                                    'fields'     => {'level' => 'debug', 'review' => 1}
                                  }
                                 ]
     shield_username          => 'host',
     shield_password          => 'secret',
     elasticsearch_proxy_host => 'elasticsearchproxy.myserver.com',
   }

Hiera data example

filebeats::prospectors:
  - input_type: 'log'
    paths:
      - '/var/log/auth.log'
    doc_type: 'auth'
  - input_type: 'log'
    paths:
      - '/var/log/my_app.log'
    doc_type: 'my_app'
  - input_type: 'log'
    paths:
      - '/var/log/security.log'
    doc_type: 'security'
    fields:
      level: 'debug'
      review: 1

Reference

• Package

Configures the apt resrouce for filebeats.

• Config

Configures the filebeats.yml file.

• Service

Ensures the service is running.

• Params

Specifies defaults for the installation and configuration

Limitations

Does not support all options available to filebeats configuration.

Development

All pull requests are welcome. This module was just created for our use and functionality will be added as we require it.

##2017-02-02 - Release 0.1.6 ###Summary

Add support for prospector fields

• Added new prospector option, fields
• Thanks for the contribution @belskiiartem

##2015-04-13 - Release 0.1.5 ###Summary

Adding logstash output

• Added new parameters for logstash output
• Added param to specify index to ship for both elasticsearch and logstash
• Added service_state param to allow for overriding
• Added loadbalancing param for logstash output

##2015-04-08 - Release 0.1.4 ###Summary

Fix documentation issues

• Minor indetation in documentation

##2015-04-08 - Release 0.1.3 ###Summary

Fix documentation issues

• Fix code examples in README, still trying to get a hang of this

##2015-04-08 - Release 0.1.2 ###Summary

Fix documentation issues

• Fix code examples in README

##2015-04-07 - Release 0.1.1 ###Summary

Minor update to include multiple prospectors

• This update allows for an array of hashes to configure multiple prospectors
• This includes setting of input_type and document type in each hash

##2015-04-06 - Release 0.1.0 ###Summary

First major release with tested (on our environment) log exporting

• Added logging options for filebeats

##2015-04-01 - Release 0.0.13 ###Summary

• Minor bug fixes
• Added TLS/SSL support
• Added protocol option

##2015-03-24 - Release 0.0.9 ###Summary

First working release of basic puppet filebeats module

• This version only included very basic funcionality for installing and configuring file beats on Debian Wheezy
• Fixed minor resource ordering issues
• Fixed some typos in the filebeat template
• Fixed service notify errors
• Added shield username and password options