puppet_metrics_dashboard

• Description
• Setup
  • Upgrade notes, breaking changes in v2
  • Determining where Telegraf runs
  • Beginning with puppet_metrics_dashboard
    • Minimal configuration
• Usage
  • To install example dashboards and use the Telegraf collection method (default)
  • Configure Telegraf for one or more masters / PuppetDB nodes / Postgres nodes
  • Enable Graphite support
  • Enable Telegraf, Graphite, and Archive (puppet_metrics)
  • Enable SSL
  • Allow access to PE-managed Postgres nodes with the following class
  • Profile defined types
    • Note on using the defined types
  • Other possibilities
• Reference
• Limitations
  • Repo failure for InfluxDB packages
  • Postgres metrics collection requires Telegraf version 1.9.1 or later
• Development

Description

This module is used to configure Grafana and InfluxDB to consume metrics from Puppet services. By default the metrics collection is done by Telegraf.

These services can all run on a single server by applying the base class. You also have the option to use the included defined types to configure Telegraf on each of your Puppet infrastructure components (master, compilers, PuppetDB, Postgres server) and the metrics will be stored on another server running Grafana and InfluxDB. In environments where there is an existing Grafana / InfluxDB instance, the later option is recommended. See Determining where Telegraf runs for further details.

You have the option of collecting metrics using any or all of these methods:

• Through Telegraf, which polls several of Puppet's metrics endpoints (recommended)
• Through Archive files from the puppetlabs/puppet_metrics_collector module
• Natively, via Puppetserver's built-in graphite support

Setup

Upgrade notes, breaking changes in v2

The puppet_metrics_dashboard::profile::postgres class is now deprecated and you should use the puppet_metrics_dashboard::profile::Master::postgres_access class instead.

Parameters telegraf_agent_interval and http_response_timeout were previously integers but are now strings. The value should match a time interval, such as 5s, 10m, or 1h.

influxdb_urls was previously a string, it should now be an array.

Previous versions of this module put several [[inputs.httpjson]] entries in /etc/telegraf/telegraf.conf. These entries should be removed now as all module-specific settings now reside in individual files within /etc/telegraf/telegraf.d/. Telegraf will continue to work if you do not remove them, however, the old [[inputs.httpjson]] will not be updated going forward.

Determining where Telegraf runs

Telegraf can run on the Grafana server or on each Puppet infrastructure node. To configure Telegraf to run on the same host that Grafana runs on, use the puppet_metrics_dashboard class and the parameters: master_list, puppetdb_list and postgres_host_list. These parameters determine which hosts that Telegraf polls.

To configure Telegraf to run on each Puppet infrastructure node, use the corresponding profiles for those hosts. See Profile defined types. The puppet_metrics_dashboard class is still applied to a separate host to setup Grafana and InfluxDB and the profile classes configure Telegraf when applied to your Puppet infrastructure hosts.

Beginning with puppet_metrics_dashboard

Minimal configuration

Configures Grafana, InfluxDB, and Telegraf, with an InfluxDB datasource and a database called "puppet_metrics"

include puppet_metrics_dashboard

Usage

To install example dashboards and use the Telegraf collection method (default)

class { 'puppet_metrics_dashboard':
  add_dashboard_examples => true,
}

add_dashboard_examples enforces state on the dashboards. Remove this later if you want to make edits to the examples or add the overwrite_dashboards parameter to disable overwriting the dashboards after the first run.

class { 'puppet_metrics_dashboard':
  add_dashboard_examples => true,
  overwrite_dashboards   => false,
}

Configure Telegraf for one or more masters / PuppetDB nodes / Postgres nodes

class { 'puppet_metrics_dashboard':
  master_list         => ['master1.com',
                          # Alternate ports may be configured using
                          # a pair of: [hostname, port_number]
                          ['master2.com', 9140]],
  puppetdb_list       => ['puppetdb1','puppetdb2'],
  postgres_host_list  => ['postgres01','postgres02'],
}

Enable Graphite support

class { 'puppet_metrics_dashboard':
  add_dashboard_examples => true,
  consume_graphite       => true,
  influxdb_database_name => ["graphite"],
  master_list            => ["master01.example.com","master02.org"],
}

• This method requires enabling on the master side as described here. The hostname(s) that you use in master_list should match the value(s) that you used for metrics_server_id in the puppet_enterprise::profile::master class.

Enable Telegraf, Graphite, and Archive (puppet_metrics)

class { 'puppet_metrics_dashboard':
  add_dashboard_examples => true,
  influxdb_database_name => ['puppet_metrics','telegraf','graphite'],
  consume_graphite       => true,
}

Enable SSL

class { 'puppet_metrics_dashboard':
  use_dashboard_ssl => true,
}

By default, this will create a set of certificates in /etc/grafana that are based on Puppet's agent certificates. You can also specify a different location by passing the variables below, but managing the certificate content or supplying your own certificates isn't yet supported.

dashboard_cert_file dashboard_cert_key

Note: Enabling SSL on Grafana will not allow for running on privileged ports such as 443. To enable this capability you can use the suggestions documented in this Grafana documentation

Allow access to PE-managed Postgres nodes with the following class

This is required for collection of Postgres metrics. The class should be applied to the master (or Postgres server if using external Postgres).

class { 'puppet_metrics_dashboard::profile::master::postgres_access':
  telegraf_host => 'grafana-server.example.com',
}

telegraf_host is optional. If you do not specify it, the class will look for a node with the puppet_metrics_dashboard class applied in PuppetDB and use the certname of the first host returned. If the PuppetDB lookup fails and you do not specify telegraf_host then the class outputs a warning.

Profile defined types

The module includes defined types that you can use with an existing Grafana implementation. See REFERENCE.md for example usage.

Note on using the defined types

Because of the way that the Telegraf module works, these examples will overwrite any configuration in telegraf.conf if it is not already puppet-managed. See the puppet-telegraf documentation on how to manage this file and add important settings.

Other possibilities

Configure the passwords for the InfluxDB and enable additional TICK Stack components.

class { 'puppet_metrics_dashboard':
  influx_db_password  => 'secret',
  grafana_http_port   => 8080,
  grafana_version     => '4.5.2',
  enable_chronograf   => true,
  enable_kapacitor    => true,
}

Reference

This module is documented via pdk bundle exec puppet strings generate --format markdown. Please see REFERENCE.md for more info.

Limitations

Repo failure for InfluxDB packages

When installing InfluxDB on CentOS/RedHat 6 or 7 you may encounter the following error message. This is due to a mismatch in the ciphers available on the local OS and on the InfluxDB repo.

Error: Execution of '/usr/bin/yum -d 0 -e 0 -y install telegraf' returned 1: Error: Cannot retrieve repository metadata (repomd.xml) for repository: influxdb. Please verify its path and try again
Error: /Stage[main]/Pe_metrics_dashboard::Telegraf/Package[telegraf]/ensure: change from purged to present failed: Execution of '/usr/bin/yum -d 0 -e 0 -y install telegraf' returned 1: Error: Cannot retrieve repository metadata (repomd.xml) for repository: influxdb. Please verify its path and try again

To rectify the issue, please update nss and curl on the affected system.

yum install curl nss --disablerepo influxdb

Postgres metrics collection requires Telegraf version 1.9.1 or later

Development

Please see CONTRIBUTING.md

Reference

Classes

Public Classes

• puppet_metrics_dashboard: Installs and configures Grafana with InfluxDB for monitoring Puppet infrastructure.
• puppet_metrics_dashboard::profile::master::postgres_access: Apply this class to a PE-managed postgres instance to allow access from telegraf
• puppet_metrics_dashboard::profile::postgres: This class is deprecated. Please use the Puppet_metrics_dashboard::Profile::Master::Postgres_access class.

Private Classes

• puppet_metrics_dashboard::config: Configures dashboard components.
• puppet_metrics_dashboard::dashboards: Configures Grafana dashboards.
• puppet_metrics_dashboard::dashboards::graphite: Installs graphite example dashboards
• puppet_metrics_dashboard::dashboards::puppet_metrics: Installs puppet_metrics example dashboards
• puppet_metrics_dashboard::dashboards::telegraf: Installs telegraf example dashboards
• puppet_metrics_dashboard::grafana: Install and configure Grafana
• puppet_metrics_dashboard::install: Installs InfluxDB components.
• puppet_metrics_dashboard::post_start_configs: InfluxDB post-start configs
• puppet_metrics_dashboard::repos: Configures InfluxDB and Grafana repos
• puppet_metrics_dashboard::service: Manages services
• puppet_metrics_dashboard::telegraf: Install and configure Telegraf
• puppet_metrics_dashboard::telegraf::config: Configures Telegraf
• puppet_metrics_dashboard::telegraf::service: Manages the Telegraf service

Defined types

• puppet_metrics_dashboard::certs: This class creates a certificates for Grafana and for connecting to PE Postgres.
• puppet_metrics_dashboard::profile::compiler: Apply this class to a master or compiler to collect puppetserver metrics
• puppet_metrics_dashboard::profile::master::postgres: Apply this class to an agent running pe-postgresql to collect postgres metrics
• puppet_metrics_dashboard::profile::puppetdb: Apply this class to a node running puppetdb to collect puppetdb metrics

Functions

• puppet_metrics_dashboard::puppetdb_metrics: function used to determine the set of needed PuppetDB metrics based on PE version

Classes

puppet_metrics_dashboard

The puppet_metrics_dashboard module installs and configures an InfluxDB stack monitor the Puppet infrastructure components.

Examples

Default Configuration

include puppet_metrics_dashboard

Configure Telegraf on a list of masters and PuppetDB servers

class { 'puppet_metrics_dashboard':
  configure_telegraf  => true,
  enable_telegraf     => true,
  master_list         => ['master1.com',
                          # Alternate ports may be configured using
                          # a list of: `[hostname, port_number]`
                          ['master2.com', 9140]],
  puppetdb_list       => ['puppetdb1',
                          ['puppetdb2', 8100]],
}

Install example dashboards for all of the collection methods

class { 'puppet_metrics_dashboard':
  add_dashboard_examples => true,
  influxdb_database_name => ['puppet_metrics','telegraf','graphite'],
}

Configure telegraf for one or more masters / puppetdb nodes

class { 'puppet_metrics_dashboard':
  configure_telegraf  => true,
  enable_telegraf     => true,
  master_list         => ['master1.com','master2.com'],
  puppetdb_list       => ['puppetdb1','puppetdb2'],
}

Enable Graphite support

class { 'puppet_metrics_dashboard':
  add_dashboard_examples => true,
  consume_graphite       => true,
  influxdb_database_name => ["graphite"],
  master_list            => ["master01.example.com","master02.org"],
}

Enable Telegraf, Graphite, and Archive

class { 'puppet_metrics_dashboard':
  add_dashboard_examples => true,
  influxdb_database_name => ['puppet_metrics','telegraf','graphite'],
  consume_graphite       => true,
  configure_telegraf     => true,
  enable_telegraf        => true,
}

Enable SSL

class { 'puppet_metrics_dashboard':
  use_dashboard_ssl => true,
}

Parameters

The following parameters are available in the puppet_metrics_dashboard class.

add_dashboard_examples

Data type: Boolean

Whether to add the Grafana dashboard example dashboards for the configured InfluxDB databases. Valid values are true, false. Defaults to false. Note: These dashboards are managed and any changes will be overwritten unless the overwrite_dashboards is set to false.

manage_repos

Data type: Boolean

Whether or not to setup yum / apt repositories for the dependent packages Valid values are true, false. Defaults to true

dashboard_cert_file

Data type: String

The location of the Grafana certficiate. Defaults to "/etc/grafana/${clientcert}_cert.pem" Only used when configuring use_dashboard_ssl is true.

dashboard_cert_key

Data type: String

The location of the Grafana private key. Defaults to "/etc/grafana/${clientcert}_key.pem" Only used when configuring use_dashboard_ssl is true.

configure_telegraf

Data type: Boolean

Whether to configure the telegraf service. Valid values are true, false. Defaults to true This parameter enables configuring telegraf to query the master_list and puppetdb_list endpoints for metrics. Metrics will be stored in the telegraf database in InfluxDb. Ensure that influxdb_database_name contains telegraf when using this parameter. Note: This parameter is only used if enable_telegraf is set to true.

consume_graphite

Data type: Boolean

Whether to enable the InfluxDB Graphite plugin. Valid values are true, false. Defaults to false This parameter enables the Graphite plugin for InfluxDB to allow for injesting Graphite metrics. Ensure influxdb_database_name contains graphite when using this parameter. Note: If using Graphite metrics from the Puppet Master, this needs to be set to true.

grafana_http_port

Data type: Integer

The port to run Grafana on. Valid values are Integers from 1024 to 65536. Defaults to 3000 The grafana port for the web interface. This should be a nonprivileged port (above 1024).

grafana_password

Data type: String

The password for the Grafana admin user. Defaults to 'admin'

grafana_version

Data type: String

The grafana version to install. Valid values are String versions of Grafana. Defaults to '4.5.2'

influxdb_database_name

Data type: Array[String]

An array of databases that should be created in InfluxDB. Valid values are 'puppet_metrics','telegraf', 'graphite', and any other string. Defaults to ['puppet_metrics'] Each database in the array will be created in InfluxDB. 'puppet_metrics','telegraf', and 'graphite' are specially named and will be used with their associated metric collection method. Any other database name will be created, but not utilized with components in this module.

influx_db_password

Data type: String

The password for the InfluxDB admin user. Defaults to 'puppet'

enable_kapacitor

Data type: Boolean

Whether to install kapacitor. Valid values are true, false. Defaults to false Install kapacitor. No configuration of kapacitor is included at this time.

enable_chronograf

Data type: Boolean

Whether to install chronograf. Valid values are true, false. Defaults to false Installs chronograf. No configuration of chronograf is included at this time.

enable_telegraf

Data type: Boolean

Whether to install telegraf. Valid values are true, false. Defaults to false Installs telegraf. No configuration is done unless the configure_telegraf parameter is set to true.

master_list

Data type: Puppet_metrics_dashboard::HostList

A list of Puppet Master servers that Telegraf will be configured to collect metrics from. Entries in the list may be:

• A single string that contains a hostname or IP address. The module will use a default port number of 8140.
• A list of length two, where the first entry is a string that contains a hostname or IP address and the second entry is an integer that specifies the port number. Defaults to [$trusted['certname']]

puppetdb_metrics

Data type: Puppet_metrics_dashboard::Puppetdb_metric

An array of hashes containing name / url pairs for each puppetdb metric. See functions/puppetdb_metrics.pp for defaults.

Default value: puppet_metrics_dashboard::puppetdb_metrics()

influxdb_urls

Data type: Array[String]

An array for telegraf's config defining where influxdb instances are

telegraf_db_name

Data type: String

The database in influxdb where telefraf metrics are stored

telegraf_agent_interval

Data type: String[2]

How often the telefraf agent queries for metrics. Defaults to "5s"

http_response_timeout

Data type: String[2]

How long to wait for the queries by telegraf to finish before giving up. Defaults to "5s"

pg_query_interval

Data type: String[2]

How often postgres queries will run when monitoring a postgres host. Defaults to "10m"

overwrite_dashboards

Data type: Boolean

Whether to overwrite the example Grafana dashboards. Valid values are true, false. Defaults to false This paramater disables overwriting the example Grafana dashboards. It takes effect after the second Puppet run and popultes the overwrite_dashboards_disabled fact. This only takes effect when add_dashboard_examples is set to true.

puppetdb_list

Data type: Puppet_metrics_dashboard::HostList

A list of PuppetDB servers that Telegraf will be configured to collect metrics from. Entries in the list may be:

• A single string that contains a hostname or IP address. The module will use a default port number of 8081.
• A list of length two, where the first entry is a string that contains a hostname or IP address and the second entry is an integer that specifies the port number. Defaults to [$trusted['certname']]

postgres_host_list

Data type: Puppet_metrics_dashboard::HostList

A list of PostgreSQL servers that Telegraf will be configured to collect metrics from. Entries in the list may be:

• A single string that contains a hostname or IP address. The module will use a default port number of 5432.
• A list of length two, where the first entry is a string that contains a hostname or IP address and the second entry is an integer that specifies the port number. Defaults to [$trusted['certname']]

use_dashboard_ssl

Data type: Boolean

Whether to enable SSL on Grafana. Valid values are true, false. Defaults to false

overwrite_dashboards_file

Data type: String

File in use to populate the overwrite_dashboards fact

influx_db_service_name

Data type: String

Name of the influxdb service for the operating system

puppet_metrics_dashboard::profile::master::postgres_access

Apply this class to a PE-managed postgres instance to allow access from telegraf

Examples

Allow access to PE-managed Postgres nodes

class { 'puppet_metrics_dashboard::profile::master::postgres_access':
  telegraf_host => 'grafana-server.example.com',
}

Parameters

The following parameters are available in the puppet_metrics_dashboard::profile::master::postgres_access class.

telegraf_host

Data type: String

The FQDN of the host where telegraf runs. Defaults to an empty string. You can explicitly set this parameter or the class attempts to lookup which host has the puppet_metrics_dashboard class applied in PuppetDB. If the parameter is not set and the lookup does not return anything we issue a warning.

Default value: ''

puppet_metrics_dashboard::profile::postgres

This class is deprecated. Please use the Puppet_metrics_dashboard::Profile::Master::Postgres_access class.

Parameters

The following parameters are available in the puppet_metrics_dashboard::profile::postgres class.

grafana_host

Data type: String

The FQDN of the host where telegraf runs. Defaults to an empty string. You can explicitly set this parameter or the class attempts to lookup which host has the puppet_metrics_dashboard class applied in PuppetDB. If the parameter is not set and the lookup does not return anything we issue a warning.

Default value: ''

Defined types

puppet_metrics_dashboard::certs

This class creates a set of certificates in /etc/${service}. These certificates are used when configuring Grafana to use SSL and to connect to PE Postgres. The certificates are based on the agent's own Puppet certificates.

Parameters

The following parameters are available in the puppet_metrics_dashboard::certs defined type.

service

Data type: Any

The service name associated with these certificates.

Default value: $name

puppet_metrics_dashboard::profile::compiler

Apply this class to a master or compiler to collect puppetserver metrics

Examples

Add telegraf to a master / compiler

puppet_metrics_dashboard::profile::compiler{ $facts['networking']['fqdn']:
  timeout => '5s',
}

Parameters

The following parameters are available in the puppet_metrics_dashboard::profile::compiler defined type.

timeout

Data type: String[2]

Default timeout of http calls. Defaults to 5 seconds

Default value: lookup('puppet_metrics_dashboard::http_response_timeout')

compiler

Data type: Variant[String,Tuple[String, Integer]]

The FQDN of the compiler / master. Defaults to the FQDN of the server where the profile is applied

Default value: $facts['networking']['fqdn']

port

Data type: Integer[1]

The port that the puppetserver service listens on on your compiler. Defaults to 8140

Default value: 8140

interval

Data type: String[2]

The frequency that telegraf will poll for metrics. Defaults to '5s'

Default value: '5s'

puppet_metrics_dashboard::profile::master::postgres

Apply this class to an agent running pe-postgresql to collect postgres metrics

Examples

Add telegraf to a postgres server

puppet_metrics_dashboard::profile::master::postgres{ $facts['networking']['fqdn']:
  query_interval => '10m',
}

Parameters

The following parameters are available in the puppet_metrics_dashboard::profile::master::postgres defined type.

query_interval

Data type: String[2]

How often to run the queries in minutes. Defaults to 10 minutes.

Default value: lookup('puppet_metrics_dashboard::pg_query_interval')

postgres_host

Data type: Variant[String,Tuple[String, Integer]]

The FQDN of the postgres host. Defaults to the FQDN of the server where the profile is applied

Default value: $facts['networking']['fqdn']

port

Data type: Integer[1]

The port that the postgres service listens on. Defaults to 5432

Default value: 5432

puppet_metrics_dashboard::profile::puppetdb

Apply this class to a node running puppetdb to collect puppetdb metrics

Examples

Add telegraf to a puppetdb node

puppet_metrics_dashboard::profile::puppetdb{ $facts['networking']['fqdn']:
  timeout          => '5s',
  puppetdb_metrics => puppet_metrics_dashboard::puppetdb_metrics(), # this is the default value
}

Parameters

The following parameters are available in the puppet_metrics_dashboard::profile::puppetdb defined type.

timeout

Data type: String[2]

Default timeout of http calls. Defaults to 5 seconds

Default value: lookup('puppet_metrics_dashboard::http_response_timeout')

puppetdb_metrics

Data type: Puppet_metrics_dashboard::Puppetdb_metric

An array of hashes containing name / url pairs for each puppetdb metric. See functions/puppetdb_metrics.pp for defaults.

Default value: puppet_metrics_dashboard::puppetdb_metrics()

puppetdb_host

Data type: Variant[String,Tuple[String, Integer]]

The FQDN of the puppetdb host. Defaults to the FQDN of the server where the profile is applied.

Default value: $facts['networking']['fqdn']

port

Data type: Integer[1]

The port that the puppetdb service listens on on your compiler. Defaults to 8081

Default value: 8081

interval

Data type: String[2]

The frequency that telegraf will poll for metrics. Defaults to '5s'

Default value: '5s'

Functions

puppet_metrics_dashboard::puppetdb_metrics

Type: Puppet Language

The list of metrics to pull from PuppetDB depends on the PE version. To avoid having a data file for each version we utilize this function to build the needed array of hashes.

puppet_metrics_dashboard::puppetdb_metrics()

The list of metrics to pull from PuppetDB depends on the PE version. To avoid having a data file for each version we utilize this function to build the needed array of hashes.

Returns: Array[Hash] An array of hashes containing name / url pairs for each puppetdb metric.

Change log

All notable changes to this project will be documented in this file. The format is based on Keep a Changelog and this project adheres to Semantic Versioning.

2019-7-11 - Release - 2.0.0

Changes

• The module now supports configuring Telegraf on each Puppet infrastructure node in addition to configuring it on the same server that runs Grafana. This is to better support end-users that already have a Grafana instance and just want to collect Puppet metrics. A set of profile classes has been added to allow for this.
• Improvements to dashboard templates for Telegraf and archive.
• Cleanup PuppetDB metrics in some versions.
• $telegraf_agent_interval and $http_response_timeout in the main class are now Strings (previously integers) they should looks like: '5s', '2m' or '1h'.
• $influxdb_urls has changed from a String to an Array. This is to support multiple Influxdb backends.
• Instead of defining all of Telegraf's metrics in a single file /etc/telegraf/telegraf.d/puppet_metrics_dashboard.conf, there will now be multiple files for each metric. An additional resource ensures that the old file is absent.
• Replaced params.pp with module level hiera data.
• Created a function for determining the needed PuppetDB metrics based on PE version.
• Removed the variable 'storage_metrics_db_queries' since it was not referenced anywhere.

Bugfixes

• Fix for Telegraf http timeouts not being set
• Fix for cases where certname != FQDN

2019-4-25 - Release - 1.1.5

Changes

• Added a metric for last successful file-sync commit
• Added puppetdb heap / status metrics
• Improve FOSS puppet support
• Allow port numbers of services to be specified

Bugfixes

• Stop creating /run/grafana on CentOS 7
• Cleanup /run/grafana spec tests for ubuntu
• Fixed grafana version req

2019-2-13 - Release - 1.1.0

Changes

• Code refactor to more standard layout
• Various CI and testing updates
• Updated apt-get / yum repo resources
• New feature: postgres metrics
• Moved telegraf config file from /etc/telegraf/telegraf.conf to /etc/telegraf/telegraf.d/puppet_metrics_dashboard.conf
• The SSL dashboard option no longer relies on puppetlabs/puppet_agent
• Tested and working on PE 2019.0.x

2018-11-30 - Release - 1.0.3

Changes:

• Minor fix for dependency versions

2018-08-03 - Release - 1.0.2

Changes:

• Added the missing license file.

2018-07-06 - Release - 1.0.1

Bugfixes

• Fixed an issue with RHEL7 where the grafana service wouldn't start after rebooting
• Fixed an issue with metadata.json where some of the URLs were incorrect

2018-07-02 - Release - 1.0.0

Initial forge release