puppet-graylog_collector

Table of Contents

1. Overview
2. Module Description - What the module does and why it is useful
3. Setup - The basics of getting started with graylog_collector
   • What graylog_collector affects
   • Setup requirements
   • Beginning with graylog_collector
4. Usage - Configuration options and additional functionality
5. Reference
   • Class: graylog_collector
   • Defined Type: graylog_collector::input
   • Defined Type: graylog_collector::output
6. Examples
7. Limitations - OS compatibility, etc.
8. Development - Guide for contributing to the module
9. Authors

Overview

Puppet module to manage Graylog Collector

You should refer to the Graylog Collector Documentation at http://docs.graylog.org/en/latest/pages/collector.html to aide you in the use of this module. Most parameters map to the configuration options.

Module Description

Manages the installation and configuration of Graylog Collector, including configuring inputs and outputs.

Setup

What graylog_collector affects

• Installs the Graylog Collector via archive or package
• Optionally manages package managment repos
• Optionally manages the user/group to run as
• Optionally provides an init script and optionally manages the service
• Manages the Graylog Collector configuration file

Setup Requirements

Requires an installation of Java, which is not managed by this module.

Try puppetlabs-java

If using Ubuntu or Debian and managing the graylog-collector repo with this module, you'll need puppetlabs-apt

Beginning with graylog_collector

1. Declare the base class and provide a server url to connect to.
2. Declare any inputs/outputs you'd like to manage.

Usage

Class: graylog_collector

The base class "entry point" for the module.

Defined Type: graylog_collector::input

Manages input configurations

Defined Type: graylog_collector::output

Manages output configurations

Private Class: graylog_collector::install::archive

Manages installing Graylog Collector from archives

Private Class: graylog_collector::config

Manages the configuration for Graylog Collector

Private Class: graylog_collector::service

Manages the Graylog Collector service

Reference

Class: graylog_collector

Refer to http://docs.graylog.org/en/latest/pages/collector.html#global-settings

Parameters

server_url

Required if enable_registration is true. Specifies the URL (including port) for the Graylog2 server.

enable_registration

Boolean. Default: true

collector_id

Default: 'file:/etc/graylog/collector/config/collector-id'

install_path

Default: '/usr/share'

Specifies the path to install Graylog Collector to.

Note: If installing via archive, this will append the version to the directory, as it's extracted from the tarball. A symlink will be created called "graylog-collector" that links to it.

config_dir

Default: '/etc/graylog/collector'

The directory for the Graylog Collector configuration file.

sysconfig_dir

Default: '/etc/default' or '/etc/sysconfig', depending on OS.

The sysconfig path. This is typically /etc/default or /etc/sysconfig

java_cmd

Default: '/usr/bin/java'

The absolute path to the java executable to use for running the Graylog Collector.

java_opts

Default: undef

Any additional Java options to pass when running Graylog Collector.

user

Default: 'root'

The user to run Graylog Collector as. Keep in mind that the collector needs access to whatever files you feed to it.

group

Default: 'root'

The group for the collector. This is used for ownership of the config files.

manage_user

Default: false

Specifies whether this module should manage the user. If you're running as root, you probably shouldn't let this module manage the user.

manage_group

Default: false

Specifies whether this module should manage the group. If you're running as root, you probably shouldn't let this module manage the group.

manage_init

Default: undef

Specifies whether this module should manage the init script or init config file.

If install_from is set to package, this manage_init defaults to false

If installing from archive, it defaults to true

manage_service

Default: true

Specifies whether this module should manage the service.

service_ensure

Default: 'running'

State of the service.

service_enable

Default: true

Enable the service on boot or not.

service_name

Default: 'graylog-collector'

The name of the service to manage if manage_service is true.

install_from

Default: depends on platform

Possible values are package and archive

Defaults to package for:

• Ubuntu 12x
• Ubuntu 14x
• Debian 8
• EL 7

Defaults to archive for everything else.

Refer to the Graylog Collector documentation for information about package repositories.

version

Default: depends on installation method

For archive, it defaults to 0.4.1. For package, it defaults to installed

The version of Graylog Collector to install.

source_url

Default: 'https://packages.graylog2.org/releases/graylog-collector/graylog-collector-0.4.0.tgz'

This is not relevant if install_from is set to package

URL to download the tarball from for installation when install_from is set to 'archive'

service_file

Absolute path to the location of the init script or service definition.

service_template

Optional custom template to use for the service file/init script.

manage_repo

Whether this module should manage the package repositories. This is only relevant if installing via package.

Defined Type: graylog_collector::input

Refer to http://docs.graylog.org/en/latest/pages/collector.html#input-settings and http://docs.graylog.org/en/latest/pages/collector.html#input-output-routing

input_name

Default: $title

The name of the input to manage

path

Default: undef

If collecting an absolute path, this should refer to it.

type

Default: 'file'

path_glob_root

Default: undef

If you're collecting a file glob (e.g. *.log), this should refer to the base directory that contains that glob.

Example: '/usr/share/tomcat/logs'

path_glob_pattern

Default: undef

The glob pattern used for collection.

Example: '.log' or '.{err,log}'

content_splitter

Default: 'NEWLINE'

How to split the content. Basically - how to separate messages. Valid options are 'NEWLINE' or 'PATTERN'

content_splitter_pattern

Default: undef

If splitting on 'PATTERN', what regex should be used to separate log messages.

Note that regex escapes must be escaped. So a normal regex pattern of \d should be \\d

charset

Default: 'UTF-8'

reader_interval

Default: '100ms'

global_message_fields

Hash.

Default: {}

Message fields allow for you to add your own fields to a log message. For example, appending the application name or some other metadata. The 'global' message fields parameter is intended for situations where you'd like to apply a global set of fields to all inputs, but allow for application-specific fields to be added. Since these are hashes, you would normally need a wrapper or some other trickery to combine a global set of fields with application-specific fields, or declare your global fields with every declaration of an input. This is useful, for example, when you'd like to set message fields with something like Hiera or in a base profile.

The global message fields will be merged with the 'message_fields' parameter.

Example:

global_message_fields {
  'fqdn'            => $::fqdn,
  'org_environment' => $::org_environment,
}

message_fields

Hash.

Default: {}

Refer to 'globalmessage_fields' above. These are fields _in addition to the 'global_message_fields'. These take precedence, however.

global_message_fields {
  'application_name' => 'tomcat',
  'app'              => 'pluto',
  'logfile'          => 'catalina.out',
}

outputs

Default: undef

Array.

The outputs to send this input to. See graylog::collector::output You can specify an output on the input or an input on the output (whoa).

Defined Type: graylog_collector::output

Refer to http://docs.graylog.org/en/latest/pages/collector.html#output-settings and http://docs.graylog.org/en/latest/pages/collector.html#input-output-routing

output_name

Default: title

The name of the output to configure. Example: 'gelf'

type

Default: gelf

The type of output. Valid options are 'gelf' and 'stdout'

host

Default: undef

If type is 'gelf', this refers to the host to send logs to. The Graylog2 server should have a GELF TCP input configured and listening.

port

Default: undef

If type is 'gelf', this refers to the port to send logs to. The Graylog2 server should have a GELF TCP input configured and listening.

client_tls

Boolean.

Default: false

Specifies whether to use TLS to connect when the output is 'gelf'

client_tls_cert_chain_file

Default: undef

client_tls_verify_cert

Boolean.

Default: true

client_queue_size

Default: '512'

client_connect_timeout

Default: '5000'

client_reconnect_delay

Default: '1000'

client_tcp_no_delay

Boolean.

Default: true

client_send_buffer_size

Default: '-1'

inputs

Array.

Default: undef

Works like 'outputs' with graylog_collector::input. What inputs should be sent to this output. You can leave this empty and specify them with the input if you'd like.

Examples

Example of declaring the common configuration in a base profile:

class profile::base::graylog {
  class { '::graylog_collector':
    server_url => '8.8.8.8:12900',
  }

  graylog_collector::output { 'gelf':
    type => 'gelf',
    host => '8.8.8.8',
    port => '12900',
  }
}

Example of managing an 'input' for a specific application:

class profile::apps::pluto {
  graylog_collector::input { 'pluto_catalina_out':
    path                     => "${catalina_base}/logs/catalina.out",
    content_splitter         => 'PATTERN',
    content_splitter_pattern => '^(\\d{4}-\\d{2}-\\d{2}|\\d{2}-\\w{3}-\\d{4})\\s\\d{1,2}:\\d{1,2}:\\d{1,2}',
    message_fields           => {
      'application_name' => 'tomcat',
      'app'              => 'pluto',
      'logfile'          => 'catalina.out',
    }
  }
}

Example of a file glob:

class profile::apps::pluto {
  graylog_collector::input { 'pluto_tomcat_logs':
    path_glob_root           => "${catalina_base}/logs",
    path_glob_pattern        => "*.{out,log,txt}",
    content_splitter         => 'PATTERN',
    content_splitter_pattern => '^(\\d{4}-\\d{2}-\\d{2}|\\d{2}-\\w{3}-\\d{4})\\s\\d{1,2}:\\d{1,2}:\\d{1,2}',
    message_fields           => {
      'application_name' => 'tomcat',
      'app'              => 'pluto',
    }
  }
}

An example of using a collector to add 'global_message_fields' to all inputs:

Graylog_collector::Input <| |> {
  outputs               => [ 'gelf' ],
  global_message_fields => {
    'fqdn'            => $::fqdn,
    'org_environment' => $::org_environment,
  }
}

Limitations

Tested on:

• CentOS 6
• CentOS 7
• Ubuntu 14.04

Development

TODO

• Extend package installation. Waiting on upstream for EL6. https://github.com/Graylog2/fpm-recipes/issues/42 Also track https://github.com/Graylog2/fpm-recipes

• Add additional operating system support

• Add testing

• Cleanups, especially around installation and service management

Contributing

Contributions are more than welcome! Reporting issues or code contributions.

1. Fork this repo
2. Do your work
3. Create a pull request

Authors and Contributors

• Josh Beard (josh@signalboxes.net)
• @cinacio
• @leonkyneur
• @hggh

2016-08-18 - Release 0.1.7

Summary

• Fix incorrect 'config_dir' namespace (@hggh PR #8)
• Tests: Allow 140 characters with Puppet Lint (@hggh PR #9)

2016-02-03 - Release 0.1.6

Summary

• Add group param to service class (@hggh PR #7)

2016-01-15 - Release 0.1.5

Summary

• Template syntax fixes: add missing = (@cinacio PR #6)
• Add missing parameters to service manifest (@cinacio PR #6)
• Fix output template to list inputs, not outputs (@cinacio PR #6)

2015-12-01 - Release 0.1.4

Summary

• Update module metadata to specify concat dependency (@leonkyneur PR #5)

2015-11-17 - Release 0.1.3

Summary

• Only require server_url if enable_registration is true (@hggh PR #4)

2015-10-19 - Release 0.1.2

Summary

• Quote input/output names

2015-10-13 - Release 0.1.1

Summary

• Fix typo in metadata (s/connector/collector)

2015-10-10 - Release 0.1.0

Summary

• Initial release