Forge Home

fluentd

Manage FluentD from gem install using defined types and resources for configuration

6,967 downloads

6,967 latest version

4.0 quality score

We run a couple of automated
scans to help you access a
module's quality. Each module is
given a score based on how well
the author has formatted their
code and documentation and
modules are also checked for
malware using VirusTotal.

Please note, the information below
is for guidance only and neither of
these methods should be considered
an endorsement by Puppet.

Version information

  • 0.1.0 (latest)
released Apr 6th 2016

Start using this module

  • r10k or Code Manager
  • Bolt
  • Manual installation
  • Direct download

Add this module to your Puppetfile:

mod 'warmfusion-fluentd', '0.1.0'
Learn more about managing modules with a Puppetfile

Add this module to your Bolt project:

bolt module add warmfusion-fluentd
Learn more about using this module with an existing project

Manually install this module globally with Puppet module tool:

puppet module install warmfusion-fluentd --version 0.1.0

Direct download is not typically how you would use a Puppet module to manage your infrastructure, but you may want to download the module in order to inspect the code.

Download

Documentation

warmfusion/fluentd — version 0.1.0 Apr 6th 2016

fluentd

Table of Contents

  1. Overview
  2. Module Description - What the module does and why it is useful
  3. Setup - The basics of getting started with fluentd
  4. Usage - Configuration options and additional functionality
  5. Todo List - What needs to be done
  6. Development - Guide for contributing to the module

Overview

Build Status

Manage a FluentD installation from the community managed Gem, and provide configuration for your environment using a set of puppet resources.

Module Description

This module attempts to provide a mechanism to manage a FluentD installation using the gem daemons rather than the td-agent packages which are not as well supported on some systems.

The code has been heavily based on the Puppet module created by mms-srf as I wanted to try and solve a few perceived problems with that implementation:

  1. Avoid use of td-agent and instead use fluentd gem directly
  2. Improve flexibility of configuration by enabling multiple nested elements and arrays without hardcoding

Setup

What fluentd affects

  • Installs the fluentd gem
  • Creates a minimum set of configuration files and directories
  • Provides resources to create additional named configuration files to be included automatically
  • Manages the fluentd service

Configuration

How to configure the fluentd agent to send data to a centralised Fluentd-Server

Install a Required Plugin

WARNING: Plugin configuration is not complete. Do not attempt to use this yet

Install your fluentd plugin. (Check here for the right plugin name.)

You can choose from a file or gem based installation.

include ::fluentd

fluentd::plugin { 'elasticsearch':
  plugin_type => 'gem',
  plugin_name => 'fluent-plugin-elasticsearch',
}

Configure a Source

Sources describe to fluentd where to obtain its data to process. This can include reading log files, opening tcp ports, running http services etc.

include ::fluentd

fluentd::source { 'apache':
  config => {
    'format'   => 'apache2',
    'path'     => '/var/log/apache2/access.log',
    'pos_file' => '/var/tmp/fluentd.pos',
    'tag'      => 'apache.access_log',
    'type'     => 'tail',
  },
}

fluentd::source { 'syslog':
  config => {
    'format'   => 'syslog',
    'path'     => '/var/log/syslog',
    'pos_file' => '/tmp/td-agent.syslog.pos',
    'tag'      => 'system.syslog',
    'type'     => 'tail',
  },
}

Match configuration

fluentd::match { 'forward':
  pattern  => '**',
  priority => '80',
  config   => {
    'type'    => 'forward',
    'servers' => [
      { 'host' => 'fluentd.example.com', 'port' => '24224' }
    ],
  },
}

Forest Plugin configuration

This is an example of having key/value configuration for nested elements other than the 'server' elements usually seen - This example is based on using the forest configuration plugin to manage the configuration of a fluentd-plugin-elasticsearch gem.

fluentd::match { 'forest-es':
  pattern  => '**',
  priority => '10',
  config   => {
    'type'    => 'forst',
    'subtype' => 'elasticsearch',
    'remove_prefix' => 'my.logs',
    'template' => [
      { 
        'host' => 'elasticsearch.example.com', 
        'port' => 9200 
        'logstash_prefix' => ${tag[1]},
      }
    ],
  },
}

Forwarder with Secondary

This is a very complicated example, based on the documentation of an out forwarder

fluentd::match { 'forwarder-safe':
  pattern  => '**',
  priority => '10',
  config   => {
    'type'  =>  'forward',
    'send_timeout'  =>  '60s',
    'recover_wait'  =>  '10s',
    'heartbeat_interval'  =>  '1s',
    'phi_threshold'  =>  '16',
    'hard_timeout'  =>  '60s',

    'server' => [
    {
       'name'  =>  'myserver1',
       'host'  =>  '192.168.1.3',
       'port'  =>  '24224',
       'weight'  =>  '60',
     },{
       'name'  =>  'myserver2',
       'host'  =>  '192.168.1.4',
       'port'  =>  '24224',
       'weight'  =>  '60',
     }
    ],
    'secondary' => {
     'type'  =>  'file',
     'path'  =>  '/var/log/fluent/forward-failed',
    }
  }
}

TODO

[ ] - Plugin management [ ] - Remove MASSIVE duplication of effort between source and match config - its basically the same thing twice

Development