heka

Installs and configures heka

Cristian Falcas

cristifalcas

22,639 downloads

5,682 latest version

3.1 quality score

Version information

  • 1.5.1 (latest)
  • 1.5.0
  • 1.4.0
  • 1.3.0
  • 1.2.1
  • 1.2.0
  • 1.1.0
  • 1.0.0
  • 0.19.0
  • 0.18.0
  • 0.16.1
  • 0.16.0
  • 0.15.0
  • 0.14.0
  • 0.13.2
  • 0.13.1
  • 0.13.0
  • 0.10.0
  • 0.8.0
  • 0.7.1
  • 0.7.0
  • 0.6.3
  • 0.6.1
  • 0.6.0
  • 0.5.0
released Oct 24th 2017
This version is compatible with:
  • Puppet Enterprise 3.x
  • Puppet >=3.7.0 <5.0.0
  • RedHat
    ,
    Fedora
    ,
    Debian
    ,
    Ubuntu

Start using this module

Tags: heka, logging

Documentation

cristifalcas/heka — version 1.5.1 Oct 24th 2017

heka

Build Status

Table of Contents

  1. Overview
  2. Module Description - What the module does and why it is useful
  3. Setup - The basics of getting started with heka
  4. Reference - An under-the-hood peek at what the module is doing and how
  5. Limitations - OS compatibility, etc.
  6. Development - Guide for contributing to the module

Overview

Heka is a tool for collecting and collating data from a number of different sources, performing "in-flight" processing of collected data, and delivering the results to any number of destinations for further analysis.

Module Description

Current version only works with heka 0.10.

If applicable, this section should have a brief description of the technology the module integrates with and what that integration enables. This section should answer the questions: "What does this module do?" and "Why would I use it?"

If your module has a range of functionality (installation, configuration, management, etc.) this is the time to mention it.

Journald forward:

The class support a parameter called journald_forward_enable.

This was added because of the PIPE signal that is sent to go programs when systemd-journald dies.

For more information read here: https://github.com/projectatomic/forward-journald

Usage:

  include ::forward_journald
  Class['forward_journald'] -> Class['heka']

Setup

You should have a repository added containing the following packages:

  • heka (main heka binary)
  • docker-forward-journald (if $forward_journald_on = true )

Beginning with heka

Install a heka server:

    include heka

Forward rsyslog to heka:

  class { 'rsyslog':
    package_status  => 'latest',
    purge_rsyslog_d => true,
    preserve_fqdn   => true,
  }

  class { 'rsyslog::client':
    log_local                 => true,
    listen_localhost          => false,
    high_precision_timestamps => true,
    spool_size                => '1g',
    server                    => 'localhost',
    port                      => 1514,
    remote_servers            => false,
    remote_type               => 'tcp',
    remote_forward_format     => 'RSYSLOG_ForwardFormat',
    rate_limit_burst          => 2000,
    rate_limit_interval       => 0,
  }
  
  include heka
  
  heka::inputs::tcpinput { 'FromRsyslog':
    address              => ':1514',
    decoder              => 'RsyslogDecoder',
    splitter             => 'split_on_newline',
    send_decode_failures => true,
    use_tls              => false,
    net                  => 'tcp4',
    keep_alive           => true,
  }

  heka::splitters::tokensplitter { 'split_on_newline': }

Forward all logs to a central logging server:

  heka::outputs::tcpoutput { 'SendToServer':
    message_matcher           => 'TRUE',
    address                   => 'heka_server.company.net:5565',
    use_tls                   => true,
    tls_cert_file             => "${::settings::ssldir}/certs/${::clientcert}.pem",
    tls_key_file              => "${::settings::ssldir}/private_keys/${::clientcert}.pem",
    tls_client_cafile         => "${::settings::ssldir}/certs/ca.pem",
    tls_client_auth           => 'RequireAndVerifyClientCert',
    tls_prefer_server_ciphers => true,
    tls_min_version           => 'TLS11',
  }

Configure a heka server:

  include heka

  heka::inputs::tcpinput { 'heka_server':
    address                   => ':5565',
    send_decode_failures      => true,
    use_tls                   => true,
    tls_cert_file             => "${::settings::ssldir}/certs/${::clientcert}.pem",
    tls_key_file              => "${::settings::ssldir}/private_keys/${::clientcert}.pem",
    tls_client_cafile         => "${::settings::ssldir}/certs/ca.pem",
    tls_client_auth           => 'RequireAndVerifyClientCert',
    tls_prefer_server_ciphers => true,
    tls_min_version           => 'TLS11',
    net     => 'tcp4',
  }

  # don't forward to any server from here
  Heka::Outputs::Tcpoutput <| title == 'SendToServer' |> {
    ensure => 'absent',
  }

Send logs to an elasticsearch instance:

  heka::outputs::elasticsearchoutput { 'es':
    message_matcher   => 'TRUE',
    encoder           => 'ESJsonEncoder',
    flush_count       => 10000,
    server            => 'http://localhost:9200',
  }

  heka::encoder::esjsonencoder { 'ESJsonEncoder':
    es_index_from_timestamp => true,
  }

Install various plugins:

  heka::outputs::dashboardoutput { 'Dashboard': }

Print all messages to stdout:

  # debug
  heka::outputs::logoutput { 'stdout_debug':
    message_matcher => 'TRUE',
    encoder         => 'rstencoder'
  }

  heka::encoder::rstencoder { 'rstencoder': }

Collect nginx logs:

  heka::inputs::logstreamerinput { 'logstreamerinput':
    decoder       => 'nginx_access',
    log_directory => '/var/log/nginx',
    file_match => 'access\.log(-)?(?P<Index>\d+)?(.gz)?',
    priority => ["^Index"]
  }

  heka::decoder::nginxaccesslogdecoder { 'nginx_access':
    log_format           => '$remote_addr - [$time_local] "$request" $status $body_bytes_sent $request_time "$http_referer" "$http_user_agent" "$cookie_JSESSIONID" ',
    type                 => 'combined',
    payload_keep         => true,
    user_agent_keep      => true,
    user_agent_transform => true
  }
  

Development

  • Fork the project
  • Commit and push until you are happy with your contribution
  • Send a pull request with a description of your changes

Templates standards:

  • bool variables should be checked with this rule if they can have undef as a default value (this allowes to set the variable to false):

    <% unless [nil, :undefined, :undef].include?(@variable_name) -%>variable_name = "<%= @variable_name -%>" <% end -%>

  • array variables:

    variable_array = [<%= [variable_array].flatten.map! {|sub| "'"+sub+"'"}.join(",") -%>]

  • objects:

    <% if @variable_name then -%> [<%= @name -%>.variable_name] <% variable_name.each do |key, value| -%> <%= key -%> = "<%= value -%>" <% end -%> <% end -%>