ca_extend

Table of Contents

1. Overview
2. Description - What the module does and why it is useful
3. Setup - The basics of getting started with this module
4. Usage - Configuration options and additional functionality
5. Reference - An under-the-hood peek at what the module is doing
6. Development - Guide for contributing to the module

Overview

This module can extend a certificate authority (CA) that's about to expire or has already expired.

A Puppet CA certificate is only valid for a finite time (a new installation of PE 2019.x / Puppet 6.x will create a 15 year CA, while earlier versions will create a 5 year CA; and upgrading does not extend the CA.), after which it expires. When a CA certificate expires, Puppet services will no longer accept any certificates signed by that CA, and your Puppet infrastructure will immediately stop working.

If your CA certificate is expiring soon (or it's already expired), you need to:

• Generate a new CA certificate using the existing CA keypair.
• Distribute the new CA certificate to agents.

This module can automate those tasks.

Description

This module is composed of Plans and Tasks to extend the expiration date of the CA certificate in Puppet Enterprise (and Puppet Open Source) and distribute that CA certificate to agents.

Note that, with Puppet Open Source, if the CA certificate is only used by the Puppet CA and no other integrations, there is no further action to take after using the two Plans. However, if it is used for other integrations (such as SSL encrypted PuppetDB traffic) then those integrations will need to have their copy of the CA certificate updated. If the CA certificate is stored in any keystores, those will also need to be updated.

The functionality of this module is composed into two Plans:

• ca_extend::extend_ca_cert
  • Extend the CA certificate and configure the primary Puppet server and any Compilers to use that extended certificate.
• ca_extend::upload_ca_cert
  • Distribute the CA certificate to agents using any transport supported by Puppet Bolt, such as ssh, winrm, or pcp.

Regardless of whether the CA certificate is expired, the extend_ca_cert plan may be used to extend its expiration date in-place and configure the primary Puppet server and any Compilers to use it.

After the CA certificate has been extended, there are two methods for distributing it to agents.

• Using the ca_extend::upload_ca_cert plan or another method to copy the CA certificate to agents.
• Manually deleting ca.pem on agents and letting them download that file as part of the next Puppet agent run. The agent will download that file only if it is absent, so it must be deleted to use this method.

There are also two complementary tasks to check the expiration date of the CA certificate or any agent certificates.

• ca_extend::check_ca_expiry
  • Checks if the CA certificate expires by a certain date. Defaults to three months from today.
• ca_extend::check_agent_expiry
  • Checks if any agent certificate expires by a certain date. Defaults to three months from today.

If the CA certificate is expiring or expired, you must extend it as soon as possible.

Setup

This module requires Puppet Bolt >= 1.2.0 on either on the primary Puppet server or a workstation with connectivity to the primary.

The installation procedure will differ depending on the version of Bolt. If possible, using Bolt >= 3.0.0 is recommended. For example, this will install the latest Bolt version on EL 7.

sudo rpm -Uvh https://yum.puppet.com/puppet-tools-release-el-7.noarch.rpm
sudo yum install puppet-bolt

The following two sections show how to install the module dependencies depending on the installed version of Bolt.

Bolt >= 1.2.0 < 3.0.0

The recommended procedure for these versions is to use a Bolt Puppetfile. From within a Boltdir, specify this module and puppetlabs-stdlib as dependencies and run bolt puppetfile install. For example:

mkdir -p ~/Boltdir
cd !$

cat >>Puppetfile <<EOF
mod 'puppetlabs-stdlib'

mod 'puppetlabs-ca_extend'
EOF

bolt puppetfile install

Bolt >= 3.0.0

The recommended procedure for these versions is to use a Bolt Project. When creating a Bolt project, specify this module and puppetlabs-stdlib as dependencies and initialize the project. For example:

sudo rpm -Uvh https://yum.puppet.com/puppet-tools-release-el-7.noarch.rpm
sudo yum install puppet-bolt

If your primary Puppet server or workstation has internet access, the project can be initialized with the needed dependencies with the following:

mkdir ca_extend
cd !$

bolt project init expiry --modules puppetlabs-stdlib,puppetlabs-ca_extend

Otherwise, if your primary Puppet server or workstation operates behind a proxy, initialize the project without the --modules option

mkdir ca_extend
cd !$

bolt project init expiry

Then edit your bolt-project.yaml to use the proxy according to the documentation. Next, add the module dependencies to bolt-project.yaml:

---
name: expiry
modules:
  - name: puppetlabs-stdlib
  - name: puppetlabs-ca_extend

Finally, install the modules.

bolt module install

See the "Usage" section for how to run the tasks and plans remotely or locally on the primary Puppet server.

Dependencies

• A Puppet Bolt >= 1.21.0
• puppetlabs-stdlib
• A base64 binary on the primary Puppet server which supports the -w flag
• bash >= 4.0 on the primary Puppet server

Configuration

Inventory

This module works best with a Bolt inventory file to allow for simultaneous uploads to *nix and Windows agents. See the Bolt documentation for how to configure an inventory file. See the REFERENCE.md for a sample inventory file.

Alternatively, you can use an ssh config file if you will only use that transport to upload the CA certificate to agents. Bolt defaults to using the ssh transport, which in turn will use ~/.ssh/config for options such as username and private-key.

PuppetDB

A convenient way to specify targets for the ca_extend::upload_ca_cert plan is by connecting Bolt to PuppetDB, after which --query can be used to specify targets. See REFERENCE.md for an example.

PCP

Note that you cannot use the Bolt pcp transport if your CA certificate has already expired, as the PXP-Agent service itself depends upon a valid CA certificate.

Usage

First, check the expiration of the Puppet agent certificate by running the following command as root on the primary Puppet server:

/opt/puppetlabs/puppet/bin/openssl x509 -in "$(/opt/puppetlabs/bin/puppet config print hostcert)" -enddate -noout

If, and only if, the notAfter date printed has already passed, then the primary Puppet server certificate has expired and must be cleaned up before the CA can be regenerated:

mkdir -p -m 0700 /var/puppetlabs/backups
(umask 0077 && tar czf "/var/puppetlabs/backups/ssl-$(date +'%Y%m%d%H%M%S')".tar.gz "$(puppet config print ssldir)")

find "$(puppet config print ssldir)" -name "$(puppet config print certname).pem" -delete

Once the expiration has been checked, the CA can be regenerated.

bolt plan run ca_extend::extend_ca_cert --targets <master_fqdn> compile_masters=<comma_separated_compile_master_fqdns> --run-as root

Note that if you are running extend_ca_cert locally on the primary Puppet server, you can avoid potential Bolt transport issues by specifying --targets local://$(hostname -f), e.g.

bolt plan run ca_extend::extend_ca_cert --targets local://$(hostname -f) --run-as root

bolt plan run ca_extend::upload_ca_cert cert=<path_to_cert> --targets <TargetSpec>

bolt task run ca_extend::check_ca_expiry --targets <TargetSpec>

bolt task run ca_extend::check_agent_expiry --targets <TargetSpec>

See REFERENCE.md for more detailed examples.

Reference

Puppet's security is based on a PKI using X.509 certificates.

This module's ca_extend::extend_ca_cert plan creates a new self-signed CA certificate using the same keypair as the prior self-signed CA. The new CA has the same:

• Keypair.
• Subject.
• Issuer.
• X509v3 Subject Key Identifier (the fingerprint of the public key).

The new CA has a different:

• Authority Key Identifier (just the serial number, since it's self-signed).
• Validity period (the point of the whole exercise).
• Signature (since we changed the serial number and validity period).

Since Puppet's services (and other services that use Puppet's PKI) validate certificates by trusting a self-signed CA and comparing its public key to the Signatures and Authority Key Identifiers of the certificates it has issued, it's possible to issue a new self-signed CA certificate based on a prior keypair without invalidating any certificates issued by the old CA. Once you've done that, it's just a matter of delivering the new CA certificate to every participant in the PKI.

Development

Puppet Labs modules on the Puppet Forge are open source projects, and community contributions are essential for keeping them great. We can’t access the huge number of platforms and myriad of hardware, software, and deployment configurations that Puppet is intended to serve. We want to keep it as easy as possible to contribute changes so that our modules work in your environment. There are a few guidelines that we need contributors to follow so that we can have a chance of keeping on top of things.

For more information, see our module contribution guide.

Reference

Configuration

ssh

Bolt defaults to using the ssh transport, which will in turn use ~/.ssh/config for options such as username and private-key.

Below is a sample config file and command.

Host pe-*
  User centos
  Port 22
  PasswordAuthentication no
  IdentityFile /home/adrian/.ssh/id_rsa-acceptance
  IdentitiesOnly yes
  LogLevel ERROR

$ bolt plan run ca_extend::upload_ca_cert cert=/tmp/ca.pem --run-as root --targets pe-agent.example.com
{
  "success": {
    "pe-agent.example.com": {
      "_output": "Uploaded '/tmp/ca.pem' to 'pe-agent.example.com:/etc/puppetlabs/puppet/ssl/certs/ca.pem'"
    }
  }
}

Inventory

See the Bolt inventory file documentation for a full reference.

Below is a sample inventory created using bolt-inventory-pdb and example commands.

$ cat pdb.yaml
---
query: "inventory[certname] {}"
groups:
- name: windows
  query: "inventory[certname] { facts.os.family = 'windows' }"
  config:
    transport: winrm
    winrm:
      user: Administrator
      password: foo
      ssl: false
- name: linux
  query: "inventory[certname] { facts.kernel = 'Linux' }"
  config:
    transport: ssh
    ssh:
      user: centos
      private-key: ~/.ssh/id_rsa-acceptance
      host-key-check: false

$ /opt/puppetlabs/bolt/bin/bolt-inventory-pdb pdb.yaml -o ~/.puppetlabs/bolt/inventory.yaml

$ cat ~/.puppetlabs/bolt/inventory.yaml
---
query: inventory[certname] {}
groups:
- name: windows
  query: inventory[certname] { facts.os.family = 'windows' }
  config:
    transport: winrm
    winrm:
      user: Administrator
      password: foo
      ssl: false
  nodes:
  - pe-agent-windows.example.com
- name: linux
  query: inventory[certname] { facts.kernel = 'Linux' }
  config:
    transport: ssh
    ssh:
      user: centos
      private-key: "~/.ssh/id_rsa-acceptance"
      host-key-check: false
  nodes:
  - pe-master.example.com
  - pe-agent.example.com
  - pe-compiler.example.com
nodes:
- pe-master.example.com
- pe-compiler.example.com
- pe-agent.example.com
- pe-agent-windows.example.com

$ bolt command run hostname --targets linux
Started on pe-master.example.com...
Started on pe-compiler.example.com...
Started on pe-agent.example.com...
Finished on pe-master.example.com:
  STDOUT:
    pe-master.example.com
Finished on pe-compiler.example.com:
  STDOUT:
    pe-compile.example.com
Finished on pe-agent.example.com:
  STDOUT:
    pe-agent.example.com
Successful on 3 nodes: pe-master.example.com,pe-agent.example.com,pe-compiler.example.com
Ran on 3 nodes in 0.62 seconds

$ bolt command run hostname --targets windows
Started on pe-agent-windows.example.com...
Finished on pe-agent-windows.example.com:
  STDOUT:
    pe-agent-windows.example.com
Successful on 1 node: pe-agent-windows.example.com
Ran on 1 node in 0.70 seconds

Plans

ca_extend::extend_ca_cert

Arguments

• master - Fully-qualified domain name of the Master acting as the Certificate Authority
• compile_masters - Optional comma-separated list of fully-qualified domain names of Compilers

Steps

• Runs the service task to stop the puppet and pe-puppetserver services on the (Primary) Master
• Runs the ca_extend::extend_ca_cert task to output the new CA certificate to a file, and return the path to the file and a base64 encoded string of its contents
• Runs the ca_extend::configure_master task to backup the ssl directory to /var/puppetlabs/backups, copy the new CA certificate in-place, and configure the Master to use that certificate
• Decodes the CA certificate's contents and outputs it to a temp file
• Uploads the new CA certificate to any Compilers and configures them to use that certificate

Output

All of the steps in this plan are critical to extending the certificate, so the plan will fail if any step fails. The output consists of Bolt logging messages and any failures of the steps involved.

Example

$ bolt plan run ca_extend::extend_ca_cert --targets pe-master.example.com compile_masters=pe-compiler.example.com --run-as root
Starting: plan ca_extend::extend_ca_cert
Starting: command 'echo "test" | base64 -w 0 - &>/dev/null' on localhost
Finished: command 'echo "test" | base64 -w 0 - &>/dev/null' with 0 failures in 0.0 sec
INFO: Stopping puppet services on pe-master.example.com
Starting: task service on pe-master.example.com
Finished: task service with 0 failures in 0.85 sec
Starting: task service on pe-master.example.com
Finished: task service with 0 failures in 1.95 sec
INFO: Extending CA certificate on pe-master.example.com
Starting: task ca_extend::extend_ca_cert on pe-master.example.com
Finished: task ca_extend::extend_ca_cert with 0 failures in 2.92 sec
INFO: Configuring pe-master.example.com to use the extended CA certificate
Starting: task ca_extend::configure_master on pe-master.example.com
Finished: task ca_extend::configure_master with 0 failures in 95.72 sec
Starting: task service on pe-master.example.com
Finished: task service with 0 failures in 1.64 sec
INFO: Stopping puppet services on compilers (pe-compiler.example.com)
INFO: Configuring compilers (pe-compiler.example.com) to use the extended CA certificate
Starting: file upload from /tmp/ca.pem to /etc/puppetlabs/puppet/ssl/certs/ca.pem on pe-compiler.example.com
Finished: file upload from /tmp/ca.pem to /etc/puppetlabs/puppet/ssl/certs/ca.pem with 0 failures in 0.59 sec
Starting: task run_agent on pe-compiler.example.com
Finished: task run_agent with 0 failures in 44.34 sec
INFO: Extended CA certificate decoded and stored at /tmp/ca.pem
INFO: Run the 'ca_extend::upload_ca_cert' plan to distribute the extended CA certificate to agents
Finished: plan ca_extend::extend_ca_cert in 148.06 sec

ca_extend::upload_ca_cert

Arguments

• cert - Location of the new CA certificate on disk.

This plan accepts any valid TargetSpec(s) specified by the --targets option.

Steps

• Collects facts from agents and separates them into groups of *nix and Windows
• Runs upload_file on each list of agents to distribute the CA certificate
• Constructs a JSON formatted object of the results of the uploads and returns it

Output

The output of this plan is a JSON object with two keys: success and failure. Each key contains any number of objects consisting of the agent certname and the output of the upload_file command.

Example

$ bolt plan run ca_extend::upload_ca_cert cert=/tmp/ca.pem --run-as root --query 'inventory { }'
Starting: plan ca_extend::upload_ca_cert
Starting: plan ca_extend::get_agent_facts
Starting: install puppet and gather facts on pe-master.example.com, pe-compiler.example.com, pe-agent.example.com, pe-agent-windows.example.com
Finished: install puppet and gather facts with 0 failures in 9.33 sec
Finished: plan ca_extend::get_agent_facts in 9.33 sec
Starting: plan facts
Starting: task facts on pe-master.example.com, pe-compiler.example.com, pe-agent.example.com, pe-agent-windows.example.com
Finished: task facts with 0 failures in 6.27 sec
Finished: plan facts in 6.31 sec
Starting: file upload from /tmp/ca.pem to /etc/puppetlabs/puppet/ssl/certs/ca.pem on pe-master.example.com, pe-compiler.example.com, pe-agent.example.com
Finished: file upload from /tmp/ca.pem to /etc/puppetlabs/puppet/ssl/certs/ca.pem with 0 failures in 0.66 sec
Starting: file upload from /tmp/ca.pem to C:\ProgramData\PuppetLabs\puppet\etc\ssl\certs\ca.pem on pe-agent-windows.example.com
Finished: file upload from /tmp/ca.pem to C:\ProgramData\PuppetLabs\puppet\etc\ssl\certs\ca.pem with 0 failures in 1.07 sec
Finished: plan ca_extend::upload_ca_cert in 17.41 sec
{
  "success": {
    "pe-master.example.com": {
      "_output": "Uploaded '/tmp/ca.pem' to 'pe-master.example.com:/etc/puppetlabs/puppet/ssl/certs/ca.pem'"
    },
    "pe-compiler.example.com": {
      "_output": "Uploaded '/tmp/ca.pem' to 'pe-compiler.example.com:/etc/puppetlabs/puppet/ssl/certs/ca.pem'"
    },
    "pe-agent.example.com": {
      "_output": "Uploaded '/tmp/ca.pem' to 'pe-agent.example.com:/etc/puppetlabs/puppet/ssl/certs/ca.pem'"
    },
    "pe-agent-windows.example.com": {
      "_output": "Uploaded '/tmp/ca.pem' to 'pe-agent-windows.example.com:C:\\ProgramData\\PuppetLabs\\puppet\\etc\\ssl\\certs\\ca.pem'"
    }
  }
}

Tasks

ca_extend::check_ca_expiry

Arguments

• cert - Optional location of CA certificate on disk to check. Defaults to /etc/puppetlabs/puppet/ssl/certs/ca.pem.
• date - Optional YYYY-MM-DD format date against which to check for expiration. Defaults to three months from today.

This task accepts any valid TargetSpec(s) specified by the --targets option. Can be run on any *nix agent node or the Master.

Steps

• Uses Unix openssl and date to determine if the CA certificate will expire.

Output

A JSON object with the status and expiration date of the CA certificate.

Example

{
  "status": "valid",
  "expiry date": "Feb 16 01:00:09 2034 GMT"
}

ca_extend::check_agent_expiry

Arguments

• date - Optional YYYY-MM-DD format date against which to check for expiration. Defaults to three months from today.

This task accepts any valid TargetSpec(s) specified by the --targets option. Should be run on the Master.

Steps

• Uses Unix openssl and date to determine if the agent certificates in /etc/puppetlabs/puppet/ssl/ca/signed/ will expire.

Output

A JSON object with keys for valid and expiring certificates.

Example

  {
    "valid": [
      "/etc/puppetlabs/puppet/ssl/ca/signed/pe-master.example.com.pem",
      "/etc/puppetlabs/puppet/ssl/ca/signed/pe-compiler.example.com.pem",
      "/etc/puppetlabs/puppet/ssl/ca/signed/pe-agent.example.com.pem",
      "/etc/puppetlabs/puppet/ssl/ca/signed/pe-agent.example.com",
      "/etc/puppetlabs/puppet/ssl/ca/signed/pe-agent-windows.example.com.pem",
    ],
    "expiring": [

    ]
  }

Changelog

All notable changes to this project will be documented in this file.

Release 0.1.0

Features

Bugfixes

Known Issues