Hiera module for Puppet

Table of Contents

1. Overview
2. Module Description - What the module does and why it is useful
3. Setup - The basics of getting started with hiera
   • What hiera affects
   • Setup requirements
   • Beginning with hiera
4. Usage - Configuration options and additional functionality
5. Reference - An under-the-hood peek at what the module is doing and how
6. Limitations - OS compatibility, etc.
7. Development - Guide for contributing to the module

Module Description

This module configures Hiera for Puppet.

Setup

What hiera affects

• Hiera yaml file
• Hiera datadir
• hiera-eyaml package
• keys/ directory for eyaml
• /etc/hiera.yaml for symlink

Setup requirements

If you are using the eyaml backend on:

• Puppet Enterprise 3.3 or earlier then you will need the puppetlabs-pe_gem module to install the eyaml gem using PE's gem command.
• Puppet Enterprise 3.7 or 3.8 then you will need the puppetlabs-pe_puppetserver_gem module.
• Puppet Enterprise 2015.x or FOSS puppetserver then you will need the puppetlabs-puppetserver_gem module.

Beginning with hiera

Declaring the class with a given hierarchy is a pretty good starting point:

This class will write out a hiera.yaml file in either /etc/puppetlabs/puppet/hiera.yaml or /etc/puppet/hiera.yaml (depending on if the node is running Puppet Enterprise or not).

class { 'hiera':
  hierarchy => [
    '%{environment}/%{calling_class}',
    '%{environment}',
    'common',
  ],
}

The resulting output in /etc/puppet/hiera.yaml:

---
:backends:
  - yaml
:logger: console
:hierarchy:
  - "%{environment}/%{calling_class}"
  - "%{environment}"
  - common

:yaml:
   :datadir: /etc/puppet/hieradata

Usage

Reference

This module will also allow you to configure different options for logger and merge_behavior. The default behavior is to set logger to console and merge behavior to native.

For details and valid options see Configuring Hiera.

class { 'hiera':
  hierarchy      => [
    '%{environment}/%{calling_class}',
    '%{environment}',
    'common',
  ],
  logger         => 'console',
  merge_behavior => 'deeper'
}

The resulting output in /etc/puppet/hiera.yaml:

---
:backends:
  - yaml
:logger: console
:hierarchy:
  - "%{environment}/%{calling_class}"
  - "%{environment}"
  - common

:yaml:
   :datadir: /etc/puppet/hieradata

:merge_behavior: deeper

Hiera-Eyaml-GPG

The default PKCS#7 encryption scheme used by hiera-eyaml is perfect if only simple encryption and decryption is needed.

However, if you are in a sizable team it helps to encrypt and decrypt data with multiple keys. This means that each team member can hold their own private key and so can the puppetmaster. Equally, each puppet master can have their own key if desired and when you need to rotate keys for either users or puppet masters, re-encrypting your files and changing the key everywhere does not need to be done in lockstep.

Requirements

Note: This module will create a /gpg sub-directory in the $keysdir.

1. The GPG keyring must be passphraseless on the on the PuppetServer(Master).

2. The GPG keyring must live in the /gpg sub-directory in the $keysdir.

3. The GPG keyring must be owned by the Puppet user. ex: pe-puppet

GPG Keyring Creation Tips

RNG-TOOLS

When generating a GPG keyring the system requires a good amount of entropy. To help generate entropy to speed up the process then rng-tools package on RHEL based systems or equivilent can be used. Note: Update the /etc/sysconfig/rngd or equivilent file to set the EXTRAOPTIONS to EXTRAOPTIONS="-r /dev/urandom -o /dev/random -t 5"

Keyring Generation

Below is a sample GPG answers file that will assist in generating a passphraseless key

cat << EOF >> /tmp/gpg_answers
%echo Generating a Puppet Hiera GPG Key
Key-Type: RSA
Key-Length: 4096
Subkey-Type: ELG-E
Subkey-Length: 4096
Name-Real: Hiera Data
Name-Comment: Hiera Data Encryption
Name-Email: puppet@$(hostname -d)
Expire-Date: 0
%no-ask-passphrase
# Do a commit here, so that we can later print "done" :-)
# %commit
# %echo done
EOF

You can then use the GPG answer file to generate your keyring within the /gpg sub-directory in the $keysdir

gpg --batch --homedir /etc/puppetlabs/code-staging/keys/gpg --gen-key /tmp/gpg_answers

Usage

class { 'hiera':
  hierarchy            => [
    'nodes/%{::clientcert}',
    'locations/%{::location}',
    'environments/%{::applicationtier}',
    'common',
  ],
  eyaml                => true,
  eyaml_gpg            => true,
  eyaml_gpg_recipients => 'sihil@example.com,gtmtech@example.com,tpoulton@example.com',
}

The resulting output in /etc/puppet/hiera.yaml:

---
:backends:
  - eyaml
  - yaml
:logger: console
:hierarchy:
  - "nodes/%{::clientcert}"
  - "locations/%{::location}"
  - "environments/%{::applicationtier}"
  - common

:yaml:
   :datadir: /etc/puppet/hieradata


:eyaml:
   :datadir: /etc/puppet/hieradata
   :pkcs7_private_key: /etc/puppet/keys/private_key.pkcs7.pem
   :pkcs7_public_key:  /etc/puppet/keys/public_key.pkcs7.pem
   :encrypt_method: "gpg"
   :gpg_gnupghome: "/etc/puppet/keys/gpg"
   :gpg_recipients: "sihil@example.com,gtmtech@example.com,tpoulton@example.com"

Classes

Public Classes

• hiera: Main class to configure hiera

Private Classes

• hiera::params: Handles variable conditionals
• hiera::eyaml: Handles eyaml configuration

Parameters

The following parameters are available for the hiera class:

• hierarchy The hiera hierarchy. Default: []

• backends The list of backends. Default: ['yaml'] If you supply a additional backend you must also supply the backend data in the backend_options hash.

• backend_options An optional hash of backend data for any backend. Each key in the hash should be the name of the backend as listed in the backends array. You can also supply additional settings for the backend by passing in a hash. By default the yaml and eyaml backend data will be added if you enable them via their respective parameters. Any options you supply for yaml and eyaml backend types will always override other parameters supplied to the hiera class for that backend.

  Example hiera data for the backend_options hash:

  backend_options:
    json:
      datadir: '/etc/puppetlabs/puppet/%{::environment}/jsondata'
    redis:
      password: clearp@ssw0rd        # if your Redis server requires authentication
      port: 6380                     # unless present, defaults to 6379
      db: 1                          # unless present, defaults to 0
      host: db.example.com           # unless present, defaults to localhost
      path: /tmp/redis.sock          # overrides port if unixsocket exists
      soft_connection_failure: true  # bypass exception if Redis server is unavailable; default is false
      separator: /                   # unless present, defaults to :
      deserialize: :json             # Try to deserialize; both :yaml and :json are supported
  

  NOTE: The backend_options must not contain symbols as keys ie :json: despite the hiera config needing symbols. The template will perform all the conversions to symbols in order for hiera to be happy. Because puppet does not use symbols there are minor annoyances when converting back and forth and merge data together.

• hiera_yaml The path to the hiera config file. Note: Due to a bug, hiera.yaml is not placed in the codedir. Your puppet.conf hiera_config setting must match the configured value; see also hiera::puppet_conf_manage Default:

  • '/etc/puppet/hiera.yaml' for Puppet Open Source
  • '/etc/puppetlabs/puppet/hiera.yaml' for Puppet Enterprise

• create_symlink Whether to create the symlink /etc/hiera.yaml Default: true

• datadir The path to the directory where hiera will look for databases. Default:

  • '/etc/puppet/hieradata' for Puppet Open Source
  • '/etc/puppetlabs/puppet/hieradata' for PE Puppet < 4
  • '/etc/puppetlabs/code/environments/%{::environment}/hieradata' for Puppet >= 4

• datadir_manage Whether to create and manage the datadir as a file resource. Default: true

• owner The owner of managed files and directories. Default:

  • 'puppet' for Puppet Open Source
  • 'pe-puppet' for Puppet Enterprise

• group The group owner of managed files and directories. Default:

  • 'puppet' for Puppet Open Source
  • 'pe-puppet' for Puppet Enterprise

• eyaml Whether to install, configure, and enable the eyaml backend. Also see the provider and master_service parameters. Default: false

• eyaml_name The name of the eyaml gem. Default: 'hiera-eyaml'

• eyaml_version The version of hiera-eyaml to install. Accepts 'installed', 'latest', '2.0.7', etc Default: undef

• eyaml_source An alternate gem source for installing hiera-eyaml. Default: undef, uses gem backend default

• eyaml_datadir The path to the directory where hiera will look for databases with the eyaml backend. Default: same as datadir

• eyaml_extension The file extension for the eyaml backend. Default: undef, backend defaults to '.eyaml'

• deep_merge_name The name of the deep_merge gem. Default: 'deep_merge'

• deep_merge_version The version of deep_merge to install. Accepts 'installed', 'latest', '2.0.7', etc. Default: undef

• deep_merge_source An alternate gem source for installing deep_merge. Default: undef, uses gem backend default

• deep_merge_options A hash of options to set in hiera.yaml for the deep merge behavior. Default: {}

• manage_package A boolean for wether the hiera package should be managed. Defaults to true on FOSS 3 but false otherwise.

• package_name Specifies the name of the hiera package. Default: 'hiera'

• package_ensure Specifies the ensure value of the hiera package. Default: 'present'

• confdir The path to Puppet's confdir. Default: $::settings::confdir which should be the following:

  • '/etc/puppet' for Puppet Open Source
  • '/etc/puppetlabs/puppet' for Puppet Enterprise

• logger Which hiera logger to use. Note: You need to manage any package/gem dependencies yourself. Default: undef, hiera defaults to 'console'

• cmdpath Search paths for command binaries, like the eyaml command. The default should cover most cases. Default: ['/opt/puppet/bin', '/usr/bin', '/usr/local/bin']

• create_keys Whether to create pkcs7 keys and manage key files for hiera-eyaml. This is useful if you need to distribute a pkcs7 key pair. Default: true

• merge_behavior Which hiera merge behavior to use. Valid values are 'native', 'deep', and 'deeper'. Deep and deeper values will install the deep_merge gem into the puppet runtime. Default: undef, hiera defaults to 'native'

• extra_config Arbitrary YAML content to append to the end of the hiera.yaml config file. This is useful for configuring backend-specific parameters. Default: ''

• keysdir Directory for hiera to manage for eyaml keys. Default: $confdir/keys Note: If using PE 2013.x+ and code-manager set the keysdir under the $confdir/code-staging directory to allow the code manager to sync the keys to all PuppetServers Example: /etc/puppetlabs/code-staging/keys

• puppet_conf_manage Whether to manage the puppet.conf hiera_config value or not. Default: true

• provider Which provider to use to install hiera-eyaml. Can be:

  • puppetserver_gem (PE 2015.x or FOSS using puppetserver)
  • pe_puppetserver_gem (PE 3.7 or 3.8)
  • pe_gem (PE pre-3.7)
  • puppet_gem (agent-only gem)
  • gem (FOSS using system ruby (ie puppetmaster)) Note: this module cannot detect FOSS puppetserver and you must pass provider => 'puppetserver_gem' for that to work. See also master_service. Default: Depends on puppet version detected as specified above.

• master_service The service name of the master to restart after package installation or hiera.yaml changes. Note: You must pass master_service => 'puppetserver' for FOSS puppetserver Default: 'pe-puppetserver' for PE 2015.x, otherwise 'puppetmaster'

Limitations

The eyaml_version parameter does not currently modify the eyaml version of the command-line gem on pe-puppetserver.

Development

Pull requests on github! If someone wrote spec tests, that would be awesome.

Changelog

All notable changes to this project will be documented in this file. Each new release typically also includes the latest Vox Pupuli modulesync defaults. These should not impact the functionality of the module.

2017-01-13 - Release 2.4.0

This is the last release with Puppet 3 support!

• Do not make files in the gnupg home executable
• Support deep_merge_options with 'deeper' merge_behavior
• Correct spelling of @merge_behavior in template
• Bump min version_requirement for Puppet + deps
• Strict variables fix for pe_server_version

2016-11-07 Release 2.3.0

• GH-128 The manage_package parameter now effects all packages.
• GH-161 Regression fix. Make eyaml be the first backend again.

2016-10-07 Release 2.2.0

• Modulesync with latest Vox Pupuli defaults
• Removes unused eyaml_gpg_keygen variable

Features

• Adds support for 3rd party backends #153

This module now supports configuration of any backend via a new backend_options parameter. This is a big deal as this opens the doors up to configuring any backend in a generic way. This is also backwards compatible with prior versions of this module.

2016-08-31 Release 2.1.2

• Modulesync with latest Vox Pupuli defaults
• Fix handling of gem_source (correctly handle URLs)

2016-08-19 Release 2.1.1

• Modulesync with latest Vox Pupuli defaults
• Fix: Replace to_yaml in hiera.yaml template (PR #134)

2016-05-21 Release 2.1.0

Note: this is the first release of the module in the voxpupuli namespace.

Features

• Add parameters to give more control over package management:
  • eyaml_name
  • eyaml_version
  • eyaml_source (deprecates gem_source parameter)
  • deep_merge_name
  • deep_merge_version
  • deep_merge_source
  • manage_package
  • package_name
  • package_source
• Add deep_merge_options parameter for passing parameters in hiera.yaml
• The merge_behavior parameter installs the deep_merge gem when needed.

Bugfixes

• Typo for master_service parameter in readme.
• Improve dependency management around packages
• Fix hierarchy parameter default on Puppet 4

2016-01-27 Release 2.0.1

Fixes

• Fix key creation when passing a custom hiera::keysdir

2016-01-26 Release 2.0.0

Changes

• eyaml keys/ directory moved from /etc/puppetlabs/code/keys to /etc/puppetlabs/puppet/keys on PE > 3.x. You should move you keys directory when upgrading.
• hiera::hiera_yaml default changed from /etc/puppetlabs/code/hiera.yaml to /etc/puppetlabs/puppet/hiera.yaml on Puppet >= 4.x. The hiera_yaml puppet config setting in puppet.conf should be updated when upgrading; see hiera::puppet_conf_manage
• hiera::datadir default changed from /etc/puppetlabs/puppet/hieradata to /etc/puppetlabs/code/environments/%{::environment}/hieradata on puppet versions >= 4. Verify that this is your prefered value when upgrading.

Features

• No longer using exec resources to install eyaml on puppet versions >= 4!
• Add hiera::puppet_conf_manage parameter to manage hiera_conf puppet.conf setting
• Add hiera::keysdir parameter for putting the keys somewhere other than $confdir/keys

Bugfixes

• Fix hiera.yaml and keys/ directory being overwritten by file sync on PE 2015.x
• Fix eyaml package provider detection on puppet versions >= 4

2016-01-08 Release 1.4.1

Bugfixes

• Fix rubocop linting
• Correct the name of the license file so the forge can find it.
• Add travis testing

2016-01-05 Release 1.4.0

Features

• Added hiera::create_symlink parameter to disable /etc/hiera.yaml creation
• Added hiera::master_service parameter to set the master service name
• Added ability to restart master service on hiera.yaml change
• Added beaker-rspec acceptance tests
• Bumped PE range to include 2015.3

Bugfixes

• Fixed bugs on PE 2015.2 when versioncmp() raised errors
• Fixed hiera.yaml output to be consistent across puppet 3 and 4
• Fixed stdlib metadata requirement.

2015-09-14 Release 1.3.2

Bugfixes

• Detect correct user on 2015.2.0
• Clean up hiera formatting.

2015-07-24 Release 1.3.1

Bugfixes

• Allow eyaml_version to be undef (the default) on PE 3.7/3.8

2015-07-23 Release 1.3.0

Features

• Add PE 3.8 support
• Add Puppet 4 support

Bugfixes

• Fix eyaml_datadir parameter default to datadir value
• Handle cmdpath on different versions of puppet

2015-03-05 Release 1.2.0

Features

• Added hiera::create_keys param to disable pkcs7 key generation with hiera-eyaml
• Added hiera::gem_source param to specify source of hiera-eyaml gem
• Added hiera::eyaml_version param to specify the version of eyaml

Bugfixes

• Change Modulefile to metadata.json
• Add validation on merge_behavior param
• Make it kind of work on PE 3.7 (pe-puppetserver must still be restarted after the gem is installed)

2014-11-21 Release 1.1.1

Bugfixes

• Correct handling of cmdpath (using an array of paths instead of hardcoded path). Also adds cmdpath parameter
• Fix permissions on private key (604 was a typo)
• Add "Managed by puppet" header

2014-10-15 Release 1.1.0

Features

• Added eyaml, eyaml_datadir, & eyaml_extension parameters to hiera class for configuring hiera-eyaml
• Added hiera::datadir_manage parameter to disable datadir management
• Added hiera::logger parameter to change logger
• Added hiera::merge_behavior parameter to change the hash merge behavior
• Added some spec tests
• Added new readme

Bugfixes

• Correct datadir regex {} matching

2014-05-01 Release 1.0.2

• Remove swap files from package

2014-03-25 Release 1.0.1

Bugfixes

• Readme tweak
• Use template instance variables to remove warnings

2014-02-27 Release 1.0.0

Features

• backends parameter for an array of hiera backends
• extra_config parameter for a string of extra yaml config

Bugfixes

• Correct the yaml formatting

2013-06-17 Release 0.3.1

Bugfixes

• Docs!

2013-06-17 Release 0.3.0

Features

• PE + POSS support

Bugfixes

• Only ensure datadir if it does not have %{.*}