Forge Home

Modules

Resources

PuppetEcosystemPuppet CommunityUpdates

puppet_metrics_collector

A Puppet module for gathering metrics from PE components
Project URLRSS FeedReport issues

138,132 downloads

1,413 latest version

4.8 quality score

We run a couple of automated
scans to help you access a
module's quality. Each module is
given a score based on how well
the author has formatted their
code and documentation and
modules are also checked for
malware using VirusTotal.

Please note, the information below
is for guidance only and neither of
these methods should be considered
an endorsement by Puppet.

Version information

  • 8.0.0 (latest)
  • 7.2.0
  • 7.1.1
  • 7.1.0
  • 7.0.5
  • 7.0.4
  • 7.0.3
  • 7.0.2
  • 7.0.1
  • 7.0.0
  • 6.6.0
  • 6.5.0
  • 6.4.1
  • 6.4.0
  • 6.3.0
  • 6.2.0
  • 6.1.1
  • 6.1.0
  • 6.0.0
  • 5.3.0
  • 5.2.0
  • 5.1.2
  • 5.1.1
  • 5.1.0
  • 5.0.1
  • 5.0.0
released Feb 26th 2019
This version is compatible with:
  • Puppet Enterprise 2023.6.x, 2023.5.x, 2023.4.x, 2023.3.x, 2023.2.x, 2023.1.x, 2023.0.x, 2021.7.x, 2021.6.x, 2021.5.x, 2021.4.x, 2021.3.x, 2021.2.x, 2021.1.x, 2021.0.x, 2019.8.x, 2019.7.x, 2019.5.x, 2019.4.x, 2019.3.x, 2019.2.x, 2019.1.x, 2019.0.x, 2018.1.x, 2017.3.x, 2017.2.x, 2017.1.x, 2016.5.x, 2016.4.x
  • Puppet >= 4.2.1
  • , , , , ,

Start using this module

  • r10k or Code Manager
  • Bolt
  • Manual installation
  • Direct download

Add this module to your Puppetfile:

mod 'puppetlabs-puppet_metrics_collector', '5.1.2'
Learn more about managing modules with a Puppetfile

Add this module to your Bolt project:

bolt module add puppetlabs-puppet_metrics_collector
Learn more about using this module with an existing project

Manually install this module globally with Puppet module tool:

puppet module install puppetlabs-puppet_metrics_collector --version 5.1.2

Direct download is not typically how you would use a Puppet module to manage your infrastructure, but you may want to download the module in order to inspect the code.

Download

Documentation

puppetlabs/puppet_metrics_collector — version 5.1.2 Feb 26th 2019

Table of Contents

Overview

This module collects metrics provided by the status endpoints of Puppet Enterprise services. The metrics can be used to identify performance issues that may be addressed by performance tuning.

Setup

Installation

Install this module with puppet module install puppetlabs-puppet_metrics_collector or add it to your Puppetfile.

To activate this module, classify your Primary Master (aka Master of Masters or MoM) with the puppet_metrics_collector class using your preferred classification method.

node 'master.example.com' {
  include puppet_metrics_collector
}

Configuration

This module automatically configures the hosts it queries by querying PuppetDB for PE Infrastructure Hosts. If there is an error with automatic configuration of hosts, refer to Manual Configuration of Hosts.

Parameters

For each Puppet Enterprise service (Puppetserver, PuppetDB, Orchestration, and ActiveMQ) there are associated <service_name>_hosts, <service_name>_ensure and <service_name>_port parameters. Refer to manifests/init.pp for details.

output_dir

String: Output directory for collected metrics. Defaults to /opt/puppetlabs/puppet-metrics-collector.

collection_frequency

Integer: How often to collect metrics, in minutes. Defaults to 5.

retention_days

Integer: How long to retain collect metrics, in days. Defaults to 90.

Usage

Grepping Metrics

You can search for useful information by performing a grep in the following format, run from inside the directory containing the metrics.

cd /opt/puppetlabs/puppet-metrics-collector
grep <metric_name> <service_name>/127.0.0.1/*.json

Since the metrics are compressed every night, you can only search metrics for the current day. To search older metrics, decompress the compressed files into a subdirectory of /tmp and run from inside that directory.

Grepping Puppetserver Metrics

Example:

grep average-free-jrubies puppetserver/127.0.0.1/*.json
puppetserver/127.0.0.1/20170404T170501Z.json:                "average-free-jrubies": 0.9950009285369501,
puppetserver/127.0.0.1/20170404T171001Z.json:                "average-free-jrubies": 0.9999444653324225,
puppetserver/127.0.0.1/20170404T171502Z.json:                "average-free-jrubies": 0.9999993830655706,

Grepping PuppetDB Metrics

Example:

grep queue_depth puppetdb/127.0.0.1/*.json
puppetdb/127.0.0.1/20170404T170501Z.json:            "queue_depth": 0,
puppetdb/127.0.0.1/20170404T171001Z.json:            "queue_depth": 0,
puppetdb/127.0.0.1/20170404T171502Z.json:            "queue_depth": 0,

Example for PE 2016.5 and older:

grep Cursor puppetdb/127.0.0.1/*.json
puppetdb/127.0.0.1/20170404T171001Z.json:          "CursorMemoryUsage": 0,
puppetdb/127.0.0.1/20170404T171001Z.json:          "CursorFull": false,
puppetdb/127.0.0.1/20170404T171001Z.json:          "CursorPercentUsage": 0,
puppetdb/127.0.0.1/20170404T171502Z.json:          "CursorMemoryUsage": 0,
puppetdb/127.0.0.1/20170404T171502Z.json:          "CursorFull": false,
puppetdb/127.0.0.1/20170404T171502Z.json:          "CursorPercentUsage": 0,
puppetdb/127.0.0.1/20170404T172002Z.json:          "CursorMemoryUsage": 0,
puppetdb/127.0.0.1/20170404T172002Z.json:          "CursorFull": false,
puppetdb/127.0.0.1/20170404T172002Z.json:          "CursorPercentUsage": 0,

Sharing Metrics Data

When working with Support, you may be asked to provide an archive of collected metrics data.

This module provides a utility script, puppet-metrics-collector to prepare metrics data for sharing.

/opt/puppetlabs/bin/puppet-metrics-collector create-tarball

This script creates a tar archive in the current working directory.

[root@master ~]# /opt/puppetlabs/bin/puppet-metrics-collector create-tarball
Metrics data tarball created at: /root/puppet-metrics-20170801T180338Z.tar.gz

Reference

Directory Layout

This module creates an output directory with one subdirectory for each Puppet Enterprise service (Puppetserver, PuppetDB, Orchestration, and ActiveMQ) that this module has been configured to collect. Each service directory has one subdirectory for each host. Each host directory contains one JSON file, collected every 5 minutes. Once per day, the metrics for each service are compressed and saved in the root of its directory.

Example:

/opt/puppetlabs/puppet-metrics-collector/puppetserver
├── 127.0.0.1
│   ├── 20170404T020001Z.json
│   ├── ...
│   ├── 20170404T170501Z.json
│   └── 20170404T171001Z.json
└── puppetserver-2017.04.04.02.00.01.tar.bz2
/opt/puppetlabs/puppet-metrics-collector/puppetdb
└── 127.0.0.1
│   ├── 20170404T020001Z.json
│   ├── ...
│   ├── 20170404T170501Z.json
│   ├── 20170404T171001Z.json
└── puppetdb-2017.04.04.02.00.01.tar.bz2

Cron Jobs

This module creates two cron jobs for each Puppet Enterprise service:

  • A cron job to collect the metrics
    • Runs as per collection_frequency
  • A cron job to compress collected metrics and delete metrics older than the retention period as per retention_days
    • Runs at randomly selected time between 12:00 AM and 3:00 AM

Example:

crontab -l
...
# Puppet Name: puppetserver_metrics_collection
*/5 * * * * /opt/puppetlabs/puppet-metrics-collector/scripts/puppetserver_metrics
# Puppet Name: puppetserver_metrics_tidy
0 2 * * * /opt/puppetlabs/puppet-metrics-collector/scripts/puppetserver_metrics_tidy

Alternate Setup

Temporary Installation

While a permanent installation is recommended, this module can be temporarily installed with the following commands.

puppet module install puppetlabs-puppet_metrics_collector --modulepath /tmp;
puppet apply -e "class { 'puppet_metrics_collector': }" --modulepath /tmp;

Manual Configuration of Hosts

If necessary, you can manually configure this module by specifying parameters via the class declaration or via Hiera data. The preferred method is via Hiera data. The following examples show you how to specify those parameters for different infrastructures, and assumes you declare this module on the Primary Master.

Monolithic Infrastructure with Compile Masters

Hiera Data Example
puppet_metrics_collector::puppetserver_hosts:
 - 'master.example.com'
 - 'compile-master-1.example.com'
 - 'compile-master-2.example.com'
puppet_metrics_collector::puppetdb_hosts:
 - 'master.example.com'
Class Declaration Example
class { 'puppet_metrics_collector':
  puppetserver_hosts => [
    'master.example.com',
    'compile-master-1.example.com',
    'compile-master-2.example.com'
  ],
  puppetdb_hosts     => ['master.example.com'],
}

Split Infrastructures without Compile Masters

Hiera Data Example
puppet_metrics_collector::puppetserver_hosts:
 - 'split-master.example.com'
puppet_metrics_collector::puppetdb_hosts:
 - 'split-puppetdb.example.com'
Class Declaration Example
class { 'puppet_metrics_collector':
  puppetserver_hosts => ['split-master.example.com'],
  puppetdb_hosts     => ['split-puppetdb.example.com'],
}

Split Infrastructure with Compile Masters

Hiera Data Example
puppet_metrics_collector::puppetserver_hosts:
 - 'split-master.example.com'
 - 'compile-master-1.example.com'
 - 'compile-master-2.example.com'
 puppet_metrics_collector::puppetdb_hosts:
  - 'split-puppetdb.example.com'
Class Definition Example
class { 'puppet_metrics_collector':
  puppetserver_hosts => [
    'split-master.example.com',
    'compile-master-1.example.com',
    'compile-master-2.example.com'
  ],
  puppetdb_hosts => ['split-puppetdb.example.com'],
}

Configuration for Distributed Metrics Collection

This option collect metrics on each PE Infrastructure Host instead of collecting metrics centrally on the MoM. This option is discouraged, but allows for the collection of metrics when the MoM cannot access the API endpoints of the other PE Infrastructure Hosts. Classify each PE Infrastructure Host with this module, specifying the following parameters.

  • When classifying a Compile Master Host, specify puppetdb_metrics_ensure => absent
  • When classifying a PuppetDB Host, specify puppetserver_metrics_ensure => absent

Using with PE 3.8

You can use this module with PE 3.8, although it requires the future parser.

If the future parser is enabled, globally or in the environment, the following can be declared in site.pp.

class { 'puppet_metrics_collector':
  output_dir => '/opt/puppet/puppet_metrics_collector'
}

Otherwise, use the following commands.

puppet module install puppetlabs-puppet_metrics_collector --modulepath /tmp;
puppet apply -e "class { 'puppet_metrics_collector' : output_dir => '/opt/puppet/puppet_metrics_collector' }"  --modulepath /tmp --parser=future