Forge Home

Modules

Puppet

Resources

PuppetEcosystemPuppet CommunityUpdates

catalog_diff

Compares two catalogs and shows the differences
Project URLRSS FeedReport issues

26,804 downloads

24,247 latest version

5.0 quality score

We run a couple of automated
scans to help you access a
module's quality. Each module is
given a score based on how well
the author has formatted their
code and documentation and
modules are also checked for
malware using VirusTotal.

Please note, the information below
is for guidance only and neither of
these methods should be considered
an endorsement by Puppet.

Version information

  • 2.3.0 (latest)
  • 2.2.1
  • 2.2.0
  • 2.1.1
  • 2.1.0
  • 2.0.0
  • 1.7.0
released Aug 18th 2020
This version is compatible with:
  • Puppet Enterprise 2019.8.x, 2019.7.x, 2019.5.x, 2019.4.x, 2019.3.x, 2019.2.x, 2019.1.x, 2019.0.x, 2018.1.x, 2017.3.x, 2017.2.x, 2016.4.x
  • Puppet >= 4.10.0 < 7.0.0
  • , , , , , ,
This module has been deprecated by its author since Jul 21st 2022.

The author has suggested puppet-catalog_diff as its replacement.

Start using this module

Documentation

camptocamp/catalog_diff — version 2.3.0 Aug 18th 2020

Puppet Catalog Diff

Puppet Forge Version Puppet Forge Downloads Build Status Coverage Status By Camptocamp

Catalog Diff

Table of Contents

  1. Overview
  2. Module Description
  3. Setup
    1. Set up node discovery
    2. Set up auth.conf
  4. Usage
    1. Multi threaded compile requests
    2. Fact search
    3. Changed depth
    4. Output report
  5. Limitations
  6. Authors
  7. Contributors
  8. See Also
    1. Upload facts to PuppetDB

Overview

A tool to compare two Puppet catalogs.

Module Description

While upgrading versions of Puppet or refactoring Puppet code you want to ensure that no unexpected changes will be made prior to committing the changes.

This tool will allow you to diff catalogs created by different versions of Puppet or different environments. This will let you gauge the impact of a change before actually touching any of your nodes.

This tool is delivered as a collection of Puppet Faces. It thus requires a Puppet installation to properly run.

The diff tool recognizes catalogs in yaml, marshall, json, or pson formats. Currently automatic generation of the catalogs is done in the pson format.

The tool can automatically compile the catalogs for both your new and older servers/environments. It can ask the master to use PuppetDB to compile the catalog for the last known environment with the last known facts. It can then validate against PuppetDB that the node is still active. This filtered list should contain only machines that have not been decommissioned in PuppetDB (important as compiling their catalogs would also reactive them and their exports otherwise).

When you are comparing between different versions of Puppet using two Master servers you are going to need to copy facts from the old Master to the new one in order to be able to compile catalogs on the new Master. This is useful when upgrading Puppet version.

To upload facts to PuppetDB on a Master see the Upload facts script.

Setup

Set up node discovery

Node discovery requires an access to the PuppetDB. You'll need either:

  • have an unencrypted access to PuppetDB (port 8080, local or proxified)
  • generate a set key and certificate signed by the Puppet CA to access the PuppetDB

Set up auth.conf

Once you have set up the discovery, you need to allow access to the "diff" node to compile the catalogs for all nodes on both your old and new masters.

On Puppet 5+, you need to edit the Puppetserver's /etc/puppetlabs/puppetserver/conf.d/auth.conf file.

In your confdir modify auth.conf to allow access to /catalog. If there is an existing reference i.e. the $1 back reference for machines to compile their own catalog then simply add another line with the certificate name of the diff machine. As mentioned this can be the new master as required.

E.g. if you're using Puppet 5, you should have something like:

{
    # Allow nodes to retrieve their own catalog
    match-request: {
        path: "^/puppet/v3/catalog/([^/]+)$"
        type: regex
        method: [get, post]
    }
    allow: ["$1","catalog-diff"]
    sort-order: 500
    name: "puppetlabs catalog"
},

If you are on Puppet 6, you can activate the certless API instead with:

{
    match-request: {
        path: "^/puppet/v4/catalog"
        type: regex
        method: [post]
    }
    allow: ["catalog-diff"]
    sort-order: 500
    name: "puppetlabs certless catalog"
},

Usage

Example:

$ puppet module install camptocamp-catalog-diff
$ puppet catalog diff \
     puppet5.example.com:8140/production puppet6.example.com:8140/production \
     --use_puppetdb \
     --filter_old_env \
     --old_catalog_from_puppetdb \
     --certless \
     --show_resource_diff \
     --content_diff \
     --ignore_parameters alias \  # Puppet6 removes lots of alias parameters
     \ #--yamldir $YAMLDIR \
     \ #--ssldir $SSLDIR \
     --changed_depth 1000 \
     --configtimeout 1000 \
     --output_report "${HOME}/lastrun-$$.json" \
     --debug \
     \ #--fact_search kernel='Darwin' \
     --threads 50

Multi threaded compile requests

You can change the number of concurrent connections to the masters by passing an interger to the --threads option. This will balence the catalogs evenly on the old and new masters. This option defaults to 10 and in testing 50 threads seemed correct for 4 masters with two load balancers.

Note: When using catalog diff to compare directories, one thread per catalog comparison will be created. However, since Ruby cannot take advantage of multiple CPUs this may be of limited use comparing local catalogs. If the 'parallel' gem is installed, then one process will be forked off per CPU on the system, allowing use of all CPUs.

Fact search

You can pass --fact_search to filter the list of nodes based on a single fact value. This currently defaults to kernel=Linux if you do not pass it. This query will be passed as a filter to the PuppetDB to retrieve the list of nodes to compare.

Changed depth

Once each catalog is compiled , it is saved to the /tmp directory on the system and the face will then automatically calculate the differences between the catalogs. Once this is complete a summary of number of nodes with changes as well as nodes whose catalog would not compile are listed. You can modify the number of nodes shown here using --changed_depth option.

Output Report

You can save the last report as json to a specific location using "--output_report" This report will contain the structured data in the format of running this command with --render-as json. An example Rakefile is provided with a docs task for converting this report to (GitHub flavored) markdown. The script above also will save the output with escaped color. If you want to view that text report run less -r lastrun-$$.log

Limitations

This code only validates the catalogs, it cannot tell you if the behavior of the providers that interpret the catalog has changed so testing is still recommended, this is just one tool to take away some of the uncertainty.

You can get some inline help with:

puppet man catalog

The reports generated by this tool can be rendered as json as well as viewed in markdown using the Rakefile in this directory. A web viewer is also available at https://github.com/camptocamp/puppet-catalog-diff-viewer

Authors

R.I.Pienaar rip@devco.net / www.devco.net / @ripienaar
Zack Smith zack@puppetlabs.com / @acidprime
Raphaël Pinson raphael.pinson@camptocamp.com / @raphink

Contributors

The list of contributors can be found at: https://github.com/camptocamp/puppet-catalog-diff/graphs/contributors.

See also

Upload facts to PuppetDB

Standalone Ruby script upload_facts.rb that is used to upload yaml files with facts to PuppetDB. This is useful when you are upgradering Puppet version and uses two different Puppet Masters for this. Then you can use this script to upload facts from the old Master to the new one. The script can also be used to just refresh the facts in PuppetDB from the old Master. These facts are required to be able to compile the catalogs on the new Master.

The script uses yaml-files in the same format as stored on the Puppet Master when real agents report their facts at the beginning of a Puppet Agent execution.

The script is developed to be executed on the Puppet Master, so the yaml-facts files should be copied to the new Master using scp or similar, preferably to the $(puppet config print vardir)/yaml/facts directory.

Then all files in the directory can be uploaded to PuppetDB by using this command:

$ ./upload_facts.rb $(puppet config print vardir)/yaml/facts/*.yaml

The script is available at https://github.com/JohnEricson/upload_facts.

It's been verified to work with uploading facts from Puppet Masters running Puppet version 3 to Masters running version 5. It uses the /puppet/v3/facts/ API which is available in version 3 and >= 5 of Puppet. This API was removed in Puppet 4 but added again in 5.