Puppet & Datadog

Description

A module to:

1. install the DataDog agent
2. to send reports of puppet runs to the Datadog service Datadog.

Requirements

Puppet >=2.7.x and <=4.2.x (we may work with newer versions, but untested). For detailed informations on compatibility, check the module page on the Puppet forge.

Installation

Install datadog_agent as a module in your Puppet master's module path.

puppet module install datadog-datadog_agent

Upgrade from previous git manual install 0.x (unreleased)

You can keep using the datadog module but it becomes legacy with the release of datadog_agent 1.0.0. Upgrade to get new features, and use the puppet forge system which is way easier for maintenance.

• Delete the datadog module rm -r /etc/puppet/modules/datadog
• Install the new module from the puppet forge puppet module install datadog-datadog_agent
• Update your manifests with the new module class, basically replace datadog by datadog_agent

For instance to deploy the elasticsearch integration

include 'datadog_agent::integrations::elasticsearch'

Usage

Once the datadog_agent module is installed on your master, there's a tiny bit of configuration that needs to be done.

1. Update the default class parameters with your API key

2. Specify the module on any nodes you wish to install the DataDog Agent.

     include datadog_agent
   

   Or assign this module using the Puppet style Parameterized class: class { 'datadog_agent': api_key => "yourkey", }

   On your Puppet master, enable reporting:

     class { 'datadog_agent':
       api_key            => "yourkey",
       puppet_run_reports => true,
     }
   

   To support reporting, your Puppet master needs to have the dogapi gem installed, to do that either run the puppet agent on your master with this configuration or install it manually with gem. Please note if on Puppet Enterprise or POSS (ie. >=3.7.0) there is a soft dependency for reporting on the puppetserver_gem module. Install with puppet module install puppetlabs-puppetserver_gem - installing manually with gem will not work.

3. Include any other integrations you want the agent to use, e.g.

   include 'datadog_agent::integrations::mongo'
   

Reporting

To enable reporting of changes to the Datadog timeline, enable the report processor on your Puppet master, and enable reporting for your clients. The clients will send a run report after each check-in back to the master.

Please specify what clients/hosts you'd like to submit puppet run reports for by setting the puppet_run_reports option to true in the node configuration manifest.

class { "datadog-agent":
    api_key => "<your_api_key>",
    puppet_run_reports => true
    # ...
}

In your Puppet master /etc/puppet/puppet.conf, add these configuration options:

[main]
# No need to modify this section
# ...

[master]
# Enable reporting to datadog
reports=datadog_reports
# If you use other reports already, just add datadog_reports at the end
# reports=store,log,datadog_reports
# ...

[agent]
# ...
pluginsync=true
report=true

And on all of your Puppet client nodes add:

[agent]
# ...
report=true

If you get

err: Could not send report:
Error 400 on SERVER: Could not autoload datadog_reports:
Class Datadog_reports is already defined in Puppet::Reports

Make sure reports=datadog_reports is defined in [master], not [main].

Step-by-step

This is the minimal set of modifications to get started. These files assume puppet 2.7.x or higher.

/etc/puppet/puppet.conf

[master]
report = true
reports = datadog_reports
pluginsync = true

[agent]
report = true
pluginsync = true

/etc/puppet/manifests/nodes.pp

node "default" {
    class { "datadog_agent":
        api_key => "INSERT YOU API KEY HERE",
    }
}
node "puppetmaster" {
    class { "datadog_agent":
        api_key            => "INSERT YOUR API KEY HERE",
        puppet_run_reports => true
    }
}

/etc/puppet/manifests/site.pp

import "nodes.pp"

Run Puppet Agent

sudo /etc/init.d/puppetmaster restart
sudo puppet agent --onetime --no-daemonize --no-splay --verbose

You should see something like:

info: Retrieving plugin
info: Caching catalog for alq-linux.dev.datadoghq.com
info: Applying configuration version '1333470114'
notice: Finished catalog run in 0.81 seconds

Verify on Datadog

Go to the Setup page and you should see this

If you click on the tile, you may reconfirm it's been automatically installed.

Search for "Puppet" in the Stream and you should see something like this:

Masterless puppet

This is a specific setup, you can use https://gist.github.com/LeoCavaille/cd412c7a9ff5caec462f to set it up.

Client Settings

Tagging client nodes

The datadog agent configuration file will be recreated from the template every puppet run. If you need to tag your nodes, add an array entry in hiera

    datadog_agent::local_tags
    - 'keyname:value'
    - 'anotherkey:%{factname}'

Here are some of the other variables that be set in the datadog_agent class to control settings in the agent:

Proxy Settings

If you need to connect to the internet through a proxy, you can set proxy_host, proxy_port, proxy_user and proxy_password.

Module Development and Testing

Clone the repo

git clone git@github.com:DataDog/puppet-datadog-agent.git
cd puppet-datadog-agent

Install dependencies

bundle install
rake lint              # Check puppet manifests with puppet-lint / Run puppet-lint
rake spec              # Run spec tests in a clean fixtures directory
rake syntax            # Syntax check Puppet manifests and templates
rake syntax:manifests  # Syntax check Puppet manifests
rake syntax:templates  # Syntax check Puppet templates
pip install pre-commit
pre-commit install

Contributing

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request

Changes

1.7.1 / 2016-06-22

Notes

• [BUFIX] Fix reversed logic in hostname_extraction option.. See #189. (Thanks @davejrt).
• [BUFIX] Fix reporting on PE and POSS. Dogapi gem required in JRuby Env. See #188.
• [BUFIX] On ubuntu manifest, agent version should be explicitly configurable. See #187.
• [BUFIX] HTTP check, name is a compulsory field. See #186.
• [BUFIX] Dogstatsd should be enabled by default. See #183.

1.7.0 / 2016-04-12

Notes

• [FEATURE] Added manifest for PGBouncer. See #175. (Thanks @mcasper).

• [FEATURE] Added manifest for Consul. See #174. (Thanks @flyinprogrammer).

• [FEATURE] Added mesos master and slave manifests for individual management. See #174. (Thanks @flyinprogrammer and @jangie).

• [FEATURE] Added option to extract the hostname from puppet hostname strings with a regex capture group. See #173. (Thanks @LeoCavaille).

• [FEATURE] Added support on multiple ports per host on Redis integration. See #169. (Thanks @fzwart).

• [FEATURE] Added support for disable_ssl_validation on Apache integration. See[#171. (Thanks @BIAndrews).

• [FEATURE] Added support for SSL, additional metrics and database connection in Mongo integration. See #164. (Thanks @bflad).

• [FEATURE] Added support for multiple instance in HTTP check. See #155. (Thanks @jniesen).

• [FEATURE] Added support for multiple new datadog.conf directives. See #79. (Thanks @obowersa).

• [FEATURE] Decouple yum repo from agent package. See #168. (Thanks @b2jrock).

• [IMPROVEMENT] Moved GPG key to its own parameter. See #158. (Thanks @davidgibbons).

• [BUFIX] Updated docker to use more current docker_daemon. See #174. (Thanks @flyinprogrammer and @jangie).

• [DEPRECATE] Deprecated old docker manifest. See #174. (Thanks @flyinprogrammer).

• [DEPRECATE] Deprecated new_tag_names in docker_daemon manifest. See #176.

• [DEPRECATE] Deprecated use_mount option in base manifest. See #174. (Thanks @flyinprogrammer).

• [CI] Improved spec and docs. See #79. (Thanks @obowersa).

• [CI] Added multiple tests for integration classes. See #145. (Thanks @kitchen).

1.6.0 / 2016-01-20

Notes

• [FEATURE] Added Puppet 4 support. See #161. (Thanks @grubernaut).

• [FEATURE] Added support for optional parameters in NTP integration. See #139. (Thanks @MartinDelta).

• [BIGFIX] Use ensure_packages(), to be more polite about apt-transport-https. See #154. (Thanks @rtyler).

• [BUGFIX] Fixed Zookeeper template. See #150 (Thanks @tuxinaut).

• [BUGFIX] Raised priority of changed event types to normal - they'll now show in Datadog UI. See #156. (Thanks @rtyler).

• [BUGFIX] Require stdlib >=4.6 (provide validate_integer()). See #161. (Thanks @mrunkel-ut).

• [CI] Testable up to puppet 4.2. See #161. (Thanks @grubernaut).

• [COSMETIC] Removing trailing whitespace. See #149. (Thanks @tuxinaut).

1.5.0 / 2015-11-13

Notes

• [FEATURE] Add generic integration configuration
• [FEATURE] Add HTTPS support for yum and apt-get
• [FEATURE] Add support for warning on missing REDIS keys.
• [FEATURE] Add support for configuring the length of REDIS slow-query queue.
• [FEATURE] Add dogstatsd forwarding configuration.
• [FEATURE] Allow skipping of SSL validation.
• [FEATURE] Allow configuration of stats histogram percentiles.
• [FEATURE] Allow disabling apt-key trusting.
• [FEATURE] Add configuration of http client.
• [FEATURE] Add support for grabbing Hiera tags.

1.4.0 / 2015-09-14

Notes

• [FEATURE] Add ganglia configuration

• [FEATURE] Add rabbitmq features for queues and vhosts

• [FEATURE] Add pre-commit hooks for yaml validation and puppet-lint

• [BUGFIX] Check for rubygems definition before attempting install

• [BUGFIX] Pin rspec-puppet version to 2.2.0 to avoid unexpected test regressions

• [BUGFIX] Fix default value for ntp offset

• [BUGFIX] Be more flexible in required version of puppetlabs/ruby

• [DOC] Improve documentation for ntp integration

• [DOC] Improve documentation for postgres integration

• [DOC] Improve documentation for contributing to the repo

1.3.0 / 2015-06-01

Notes

• [FEATURE] Add collect_ec2_tags and collect_instance_metadata options to the main class

• [FEATURE] Add sock parameter in MySQL integration

• [FEATURE] Add support for graphite listener option in the main class

• [FEATURE] Add NTP integration

• [FEATURE] Add support for dogstreams array in the main class

• [FEATURE] Add HAProxy integration

• [FEATURE] Add RabbitMQ integration

• [FEATURE] Add support for an extra template appended to datadog.conf

• [FEATURE] Add Mesos integration

• [FEATURE] Add Marathon integration

• [FEATURE] Add more flexiblity to configure the docker integration

• [BUGFIX] Fix discrepancy of exact_match default in the process check compared to dd-agent

• [BUGFIX] Fix ordering of resources when installing agent

• [CI] Test on a variety of puppet & ruby versions

• [CI] Move to Travis docker infra and add some bundle caching

1.2.0 / 2015-02-24

Notes

• [FEATURE] Add zookeeper integration

• [FEATURE] Make redhat/yum base URL configurable

• [FEATURE] Add docker integration

• [FEATURE] Add postgres integration

• [FEATURE] Add use_mount option in the base datadog_agent class

• [FEATURE] Add proxy options in the base datadog_agent class

• [BUGFIX] Use correct JMX-styled tags in JMX integrations

  Careful this means that you probably have to update a buggy array of tags (that gives you nothing in the agent) to a hash of tags.

• [BUGFIX] Fix ordering in YAML templates using to_yaml broken because of ruby 1.8

• [CI] Add boilerpate for specs and linting rake tasks

• [CI] Add a travis build!

• [CI] All base manifests should have specs

1.1.1 / 2014-10-03

Notes

• [FEATURE] Expose log_to_syslog in datadog_agent class
• [BUGFIX] Fix Mongo integration YAML file generation when using tags

1.1.0 / 2014-09-22

Notes

• [FEATURE] Add facts_to_tags to the main class, to tag with facts out of the box
• [FEATURE] Add classes for Tomcat & Solr integrations
• [FEATURE] Make service_ensure and service_enable configurable allowing specific use like image builds
• [BUGFIX] Removed datadog-agent-base removal during installation that could cause yum to uninstall datadog-agent
• [BUGFIX] Fixed deprecation warning on the datadog.conf template

1.0.1