pe_to_mqtt

A Puppet module that is used for setting up a fact terminus and/or report processor for publishing fact and/or report data respectively to an MQTT broker via the MQTT protocol.

Note: This module has been tested to send data to an EMQX MQTT broker, but should work with other MQTT brokers as well as it is just making using a generic MQTT client.

Table of Contents

1. Description
2. Setup - The basics of getting started with pe_to_mqtt
   • What pe_to_mqtt affects
   • Setup requirements
   • Beginning with pe_to_mqtt
3. Usage - Configuration options and additional functionality
4. Limitations
5. Development - Contributing to the module

Description

This Puppet module can be used to configure a Puppet Enterprise server to send data to a given MQTT broker via the MQTT protocol.

• Facts data for nodes will by default be published to the topic puppet/facts
• Reports for nodes will by default be published to the topic puppet/reports

Setup

What pe_to_mqtt affects

This module will manage/install the mqtt gem on the Puppet server and will configure the following configuration files:

• puppet.conf file to make use of the fact indirector code (via a custom route_file) and report processor code installed via this module.
• mqtt_routes.yaml a custom route_file for setting the facts terminus and facts cache terminus
• pe_mqtt.yaml a configuration file for setting MQTT server details and controlling fact/report data sent to the MQTT broker

Setup Requirements

• Puppet Enterprise (PE)
• MQTT Broker (Such as EMQX. A Puppet module for configuring EMQX can be found here)

Beginning with pe_to_mqtt

Basic configuration to set up both the facts terminus and report processor to report to a given MQTT broker, classify the Puppet Primary server with the following:

class { 'pe_to_mqtt':
  mqtt_hostname => '<mqtt hostname>',
}

Usage

This module supports the use of Hiera data for setting parameters. The following is a list of parameters configurable in Hiera: (Please refer to REFERENCE.md for more details)

---
pe_to_mqtt::manage_mqtt_gem: true
pe_to_mqtt::configure_report_processor: true
pe_to_mqtt::manage_route_file: true
pe_to_mqtt::mqtt_hostname: 'mqtt.example.com'
pe_to_mqtt::mqtt_port: 1883
pe_to_mqtt::disable_report_mqtt: false
pe_to_mqtt::disable_facts_mqtt: false
pe_to_mqtt::report_publish_status: 'all'
pe_to_mqtt::report_mqtt_topic: 'puppet/reports'
pe_to_mqtt::facts_terminus: mqtt
pe_to_mqtt::facts_cache_terminus: json
pe_to_mqtt::facts_mqtt_topic: 'puppet/facts'

Common usage:

Classify a node with pe_to_mqtt:

include pe_to_mqtt

Configure pe_to_mqtt to only configure the fact terminus:

---
pe_to_mqtt::configure_report_processor: false
pe_to_mqtt::mqtt_hostname: 'mqtt.example.com'
pe_to_mqtt::mqtt_port: 1883

Configure pe_to_mqtt to only configure the fact terminus:

---
pe_to_mqtt::mqtt_hostname: 'mqtt.example.com'
pe_to_mqtt::mqtt_port: 1883
pe_to_mqtt::manage_route_file: false

Configure pe_to_mqtt to only configure the report processor:

---
pe_to_mqtt::configure_report_processor: false
pe_to_mqtt::mqtt_hostname: 'mqtt.example.com'
pe_to_mqtt::mqtt_port: 1883

Configure pe_to_mqtt to only send failed reports to MQTT broker:

---
pe_to_mqtt::mqtt_hostname: 'mqtt.example.com'
pe_to_mqtt::mqtt_port: 1883
pe_to_mqtt::report_publish_status: 'failed'

Configure pe_to_mqtt to only send reports to a different MQTT topic:

---
pe_to_mqtt::mqtt_hostname: 'mqtt.example.com'
pe_to_mqtt::mqtt_port: 1883
pe_to_mqtt::report_mqtt_topic: 'custom_topic/puppet_reports'

Configure pe_to_mqtt to only send select fields of a Puppet report to the MQTT broker (refer to REFERENCE.md for available fields to select or Puppet gem documentation4):

---
pe_to_mqtt::mqtt_hostname: 'mqtt.example.com'
pe_to_mqtt::mqtt_port: 1883
pe_to_mqtt::report_selected_fields:
  - host
  - time
  - configuration_version
  - transaction_uuid
  - report_format
  - puppet_version
  - status
  - transaction_completed
  - noop
  - noop_pending
  - environment

Configure pe_to_mqtt to only send select facts to the MQTT Broker:

---
pe_to_mqtt::mqtt_hostname: 'mqtt.example.com'
pe_to_mqtt::mqtt_port: 1883
pe_to_mqtt::facts_selected_facts:
  - os
  - memory
  - puppetversion
  - system_uptime
  - load_averages
  - ipaddress
  - fqdn

Limitations

• Currently only tested on a Puppet Enterprise Installation (PE 2021.7.3)
• Only basic configuration is available for MQTT, (e.g. SSL not yet supported).
• There can be some issues publishing facts/report data that is > 500 KB. This behaviour has been noticed for Puppet Primary Server/Compiler reports where reports published to the MQTT Broker seem to get lost when they are greater than 500 KB on EMQX. This is likely to be due to settings on the MQTT broker regarding max size limits for message payloads.

Development

If you would like to contribute with the development of this module, please feel free to log development changes in the issues register for this project

Reference

Table of Contents

Classes

• pe_to_mqtt: A class that configures the report_mqtt Puppet report processor and/or mqtt terminus on Puppetservers

Classes

pe_to_mqtt

Configures an mqtt report processor and/or mqtt fact terminus on Puppet servers

Examples

include pe_to_mqtt

Parameters

The following parameters are available in the pe_to_mqtt class:

• manage_mqtt_gem
• configure_report_processor
• manage_route_file
• mqtt_hostname
• mqtt_port
• disable_report_mqtt
• report_publish_status
• report_mqtt_topic
• report_selected_fields
• disable_facts_mqtt
• facts_terminus
• facts_cache_terminus
• facts_mqtt_topic
• facts_selected_facts

manage_mqtt_gem

Data type: Boolean

Boolean for setting whether to manage the required mqtt gem on Puppet server.

Default value: true

configure_report_processor

Data type: Boolean

Boolean for setting whether to configure puppet.conf to make use of mqtt report processor.

Default value: true

manage_route_file

Data type: Boolean

Boolean for setting whether to create a new route_file and update puppet.conf to use it.

Default value: true

mqtt_hostname

Data type: String

Hostname of the mqtt broker.

Default value: 'mqtt.example.com'

mqtt_port

Data type: Integer

Port for mqtt client connection to MQTT broker.

Default value: 1883

disable_report_mqtt

Data type: Boolean

Parameter to disable the mqtt report processor from publishing to mqtt.

Default value: false

report_publish_status

Data type: Enum['all','changed','unchanged','failed']

Parameter for controlling which reports to publish based on report status.

Default value: 'all'

report_mqtt_topic

Data type: String

MQTT topic to publish reports to.

Default value: 'puppet/reports'

report_selected_fields

Data type:

Optional[Array[Enum[
        'host',
        'time',
        'configuration_version',
        'transaction_uuid',
        'report_format',
        'puppet_version',
        'status',
        'transaction_completed',
        'noop',
        'noop_pending',
        'environment',
        'logs',
        'metrics',
        'resource_statuses',
        'corrective_change',
        'server_used',
        'catalog_uuid',
        'code_id',
        'job_id',
        'cached_catalog_status',
  ]]]

An optional array of facts to select. If not defined, all facts will be selected.

Default value: undef

disable_facts_mqtt

Data type: Boolean

Parameter for controlling which reports to publish based on report status.

Default value: false

facts_terminus

Data type: String

The mqtt facts terminus to set in route_file.

Default value: 'mqtt'

facts_cache_terminus

Data type: String

The mqtt facts cache terminus to set in route_file.

Default value: 'json'

facts_mqtt_topic

Data type: String

MQTT topic to publish facts to.

Default value: 'puppet/facts'

facts_selected_facts

Data type: Optional[Array[String[1]]]

An optional array of facts to select. If not defined, all facts will be selected.

Default value: undef

Changelog

All notable changes to this project will be documented in this file.

v0.2.1

Features

Bugfixes

• Fix error while running Puppet node clean/purge #14
• Added logic to prevent Puppet runs from failing when MQTT broker is unavailable

Known Issues

v0.2.0

Features

• Added ability to filter fields in Puppet report sent to MQTT Broker
• Added ability to filter facts sent to MQTT Broker
• Updated README limitation notes regarding payloads >500KB getting lost being due to MQTT broker settings rather than module implementation

Bugfixes

Known Issues

0.1.0

Features

• Initial Release - Basic configuration of MQTT based fact terminus and report processor

Bugfixes

Known Issues

• There may be some issues publishing facts/report data that is > 500 KB. This behaviour has been noticed for Puppet Primary Server/Compiler reports where reports published to the MQTT Broker seem to get lost when they are greater than 500 KB. (Future plans to add settings for controlling the report payload in the future which can help control payload size)