Forge Home

Modules

Puppet

Resources

PuppetEcosystemPuppet CommunityUpdates

heka

Installs and configures heka
Project URLRSS FeedReport issues

26,876 downloads

7,835 latest version

3.1 quality score

We run a couple of automated
scans to help you access a
module's quality. Each module is
given a score based on how well
the author has formatted their
code and documentation and
modules are also checked for
malware using VirusTotal.

Please note, the information below
is for guidance only and neither of
these methods should be considered
an endorsement by Puppet.

Support the Puppet Community by contributing to this module

You are welcome to contribute to this module by suggesting new features, currency updates, or fixes. Every contribution is valuable to help ensure that the module remains compatible with the latest Puppet versions and continues to meet community needs. Complete the following steps:

  1. Review the module’s contribution guidelines and any licenses. Ensure that your planned contribution aligns with the author’s standards and any legal requirements.
  2. Fork the repository on GitHub, make changes on a branch of your fork, and submit a pull request. The pull request must clearly document your proposed change.

For questions about updating the module, contact the module’s author.

Version information

  • 1.5.1 (latest)
  • 1.5.0
  • 1.4.0
  • 1.3.0
  • 1.2.1
  • 1.2.0
  • 1.1.0
  • 1.0.0
  • 0.19.0
  • 0.18.0
  • 0.16.1
  • 0.16.0
  • 0.15.0
  • 0.14.0
  • 0.13.2
  • 0.13.1
  • 0.13.0
  • 0.10.0
  • 0.8.0
  • 0.7.1
  • 0.7.0
  • 0.6.3
  • 0.6.1
  • 0.6.0
  • 0.5.0 (deleted)
released Oct 24th 2017
This version is compatible with:
  • Puppet Enterprise 3.x
  • Puppet >=3.7.0 <5.0.0
  • , , ,

Start using this module

  • r10k or Code Manager
  • Bolt
  • Manual installation
  • Direct download

Add this module to your Puppetfile:

mod 'cristifalcas-heka', '1.5.1'
Learn more about managing modules with a Puppetfile

Add this module to your Bolt project:

bolt module add cristifalcas-heka
Learn more about using this module with an existing project

Manually install this module globally with Puppet module tool:

puppet module install cristifalcas-heka --version 1.5.1

Direct download is not typically how you would use a Puppet module to manage your infrastructure, but you may want to download the module in order to inspect the code.

Download
Tags: logging, heka

Documentation

cristifalcas/heka — version 1.5.1 Oct 24th 2017

heka

Build Status

Table of Contents

  1. Overview
  2. Module Description - What the module does and why it is useful
  3. Setup - The basics of getting started with heka
  4. Reference - An under-the-hood peek at what the module is doing and how
  5. Limitations - OS compatibility, etc.
  6. Development - Guide for contributing to the module

Overview

Heka is a tool for collecting and collating data from a number of different sources, performing "in-flight" processing of collected data, and delivering the results to any number of destinations for further analysis.

Module Description

Current version only works with heka 0.10.

If applicable, this section should have a brief description of the technology the module integrates with and what that integration enables. This section should answer the questions: "What does this module do?" and "Why would I use it?"

If your module has a range of functionality (installation, configuration, management, etc.) this is the time to mention it.

Journald forward:

The class support a parameter called journald_forward_enable.

This was added because of the PIPE signal that is sent to go programs when systemd-journald dies.

For more information read here: https://github.com/projectatomic/forward-journald

Usage:

  include ::forward_journald
  Class['forward_journald'] -> Class['heka']

Setup

You should have a repository added containing the following packages:

  • heka (main heka binary)
  • docker-forward-journald (if $forward_journald_on = true )

Beginning with heka

Install a heka server:

    include heka

Forward rsyslog to heka:

  class { 'rsyslog':
    package_status  => 'latest',
    purge_rsyslog_d => true,
    preserve_fqdn   => true,
  }

  class { 'rsyslog::client':
    log_local                 => true,
    listen_localhost          => false,
    high_precision_timestamps => true,
    spool_size                => '1g',
    server                    => 'localhost',
    port                      => 1514,
    remote_servers            => false,
    remote_type               => 'tcp',
    remote_forward_format     => 'RSYSLOG_ForwardFormat',
    rate_limit_burst          => 2000,
    rate_limit_interval       => 0,
  }
  
  include heka
  
  heka::inputs::tcpinput { 'FromRsyslog':
    address              => ':1514',
    decoder              => 'RsyslogDecoder',
    splitter             => 'split_on_newline',
    send_decode_failures => true,
    use_tls              => false,
    net                  => 'tcp4',
    keep_alive           => true,
  }

  heka::splitters::tokensplitter { 'split_on_newline': }

Forward all logs to a central logging server:

  heka::outputs::tcpoutput { 'SendToServer':
    message_matcher           => 'TRUE',
    address                   => 'heka_server.company.net:5565',
    use_tls                   => true,
    tls_cert_file             => "${::settings::ssldir}/certs/${::clientcert}.pem",
    tls_key_file              => "${::settings::ssldir}/private_keys/${::clientcert}.pem",
    tls_client_cafile         => "${::settings::ssldir}/certs/ca.pem",
    tls_client_auth           => 'RequireAndVerifyClientCert',
    tls_prefer_server_ciphers => true,
    tls_min_version           => 'TLS11',
  }

Configure a heka server:

  include heka

  heka::inputs::tcpinput { 'heka_server':
    address                   => ':5565',
    send_decode_failures      => true,
    use_tls                   => true,
    tls_cert_file             => "${::settings::ssldir}/certs/${::clientcert}.pem",
    tls_key_file              => "${::settings::ssldir}/private_keys/${::clientcert}.pem",
    tls_client_cafile         => "${::settings::ssldir}/certs/ca.pem",
    tls_client_auth           => 'RequireAndVerifyClientCert',
    tls_prefer_server_ciphers => true,
    tls_min_version           => 'TLS11',
    net     => 'tcp4',
  }

  # don't forward to any server from here
  Heka::Outputs::Tcpoutput <| title == 'SendToServer' |> {
    ensure => 'absent',
  }

Send logs to an elasticsearch instance:

  heka::outputs::elasticsearchoutput { 'es':
    message_matcher   => 'TRUE',
    encoder           => 'ESJsonEncoder',
    flush_count       => 10000,
    server            => 'http://localhost:9200',
  }

  heka::encoder::esjsonencoder { 'ESJsonEncoder':
    es_index_from_timestamp => true,
  }

Install various plugins:

  heka::outputs::dashboardoutput { 'Dashboard': }

Print all messages to stdout:

  # debug
  heka::outputs::logoutput { 'stdout_debug':
    message_matcher => 'TRUE',
    encoder         => 'rstencoder'
  }

  heka::encoder::rstencoder { 'rstencoder': }

Collect nginx logs:

  heka::inputs::logstreamerinput { 'logstreamerinput':
    decoder       => 'nginx_access',
    log_directory => '/var/log/nginx',
    file_match => 'access\.log(-)?(?P<Index>\d+)?(.gz)?',
    priority => ["^Index"]
  }

  heka::decoder::nginxaccesslogdecoder { 'nginx_access':
    log_format           => '$remote_addr - [$time_local] "$request" $status $body_bytes_sent $request_time "$http_referer" "$http_user_agent" "$cookie_JSESSIONID" ',
    type                 => 'combined',
    payload_keep         => true,
    user_agent_keep      => true,
    user_agent_transform => true
  }

Development

  • Fork the project
  • Commit and push until you are happy with your contribution
  • Send a pull request with a description of your changes

Templates standards:

  • bool variables should be checked with this rule if they can have undef as a default value (this allowes to set the variable to false):

    <% unless [nil, :undefined, :undef].include?(@variable_name) -%>variable_name = "<%= @variable_name -%>" <% end -%>

  • array variables:

    variable_array = [<%= [variable_array].flatten.map! {|sub| "'"+sub+"'"}.join(",") -%>]

  • objects:

    <% if @variable_name then -%> [<%= @name -%>.variable_name] <% variable_name.each do |key, value| -%> <%= key -%> = "<%= value -%>" <% end -%> <% end -%>