data_entitlement

This module will help you setup HDP's report processor on a PE Primary or Compiler. It will also help you setup a node to host the HDP application stack.

• Description
• Setup
  • What data_entitlement affects
  • Setup Requirements
• Usage
• Reference
• Changelog

Description

There are two parts to getting started with HDP:

1. Setting up a node to run HDP itself (data_entitlement::app_stack)
2. Configuring your PE Master and any Compilers to send data to HDP (data_entitlement::report_processor)

Setup

What data_entitlement affects

This module will modify the puppet.conf configuration of any master or compiler that it is applied to. Additionally, it will install and configure Docker on the node running HDP.

Setup Requirements

HDP only works with Puppet Enterprise.

Usage

See REFERENCE.md for example usage.

Reference

A custom fact named data_entitlement-health is included as part of this module. It is a structured fact that returns information about the currently running instance of HDP. Also, if this module is installed on a node, an data_entitlement fact is included that will collect unmanaged resource information, but not land in PuppetDB.

This module is documented via pdk bundle exec puppet strings generate --format markdown. Please see REFERENCE.md for more info.

Changelog

CHANGELOG.md is generated prior to each release via pdk bundle exec rake changelog. This process relies on labels that are applied to each pull request.

Reference

Table of Contents

Classes

Public Classes

• data_entitlement::app_stack: This class takes care of configuring a node to run HDP.
• data_entitlement::data_processor: Simple class to enable the HDP data processor
• data_entitlement::proxy: This class takes care of configuring a node to run an HDP Proxy/Gateway.
• data_entitlement::resource_collector: This class adds module to the node, which adds our custom facts. Without it, we don't have the ability to pull all ral resources

Private Classes

• data_entitlement::app_stack::config
• data_entitlement::app_stack::install
• data_entitlement::app_stack::service
• data_entitlement::proxy::config
• data_entitlement::proxy::install
• data_entitlement::proxy::service

Data types

• Data_entitlement::Url: Data_entitlement::Url is a metatype that supports both single and multiple urls

Tasks

• get_service_status: Get the status of the HDP docker-compose services
• update_all_images: Automates the dance needed to get docker-compose to update the container for every service
• use_updated_image: Automates the dance needed to get docker-compose to use an updated image

Classes

data_entitlement::app_stack

This class takes care of configuring a node to run HDP.

Examples

Configure via Hiera

include data_entitlement::app_stack

Manage the docker group elsewhere

realize(Group['docker'])

class { 'data_entitlement::app_stack':
  dns_name            => 'http://data_entitlement-app.example.com',
  create_docker_group => false,
  require             => Group['docker'],
}

Enable TLS using puppet-managed certs on the frontend

class { 'data_entitlement::app_stack':
  dns_name     => 'http://data_entitlement-app.example.com',
  ui_use_tls   => true,
  ui_cert_files_puppet_managed => true,
  ui_key_file  => $profile::ssl::data_entitlement_keyfile,
  ui_cert_file => $profile::ssl::data_entitlement_full_chain,
}

Enable TLS using manually managed certs on the frontend

class { 'data_entitlement::app_stack':
  dns_name                     => 'http://data_entitlement-app.example.com',
  ui_use_tls                   => true,
  ui_cert_files_puppet_managed => false,
  ui_key_file                  => '/etc/pki/private/data_entitlement-app.key',
  ui_cert_file                 => '/etc/pki/certs/full-chain.crt',
}

Parameters

The following parameters are available in the data_entitlement::app_stack class:

• create_docker_group
• manage_docker
• log_driver
• data_dir
• data_entitlement_port
• data_entitlement_query_port
• data_entitlement_query_auth
• data_entitlement_query_username
• data_entitlement_query_password
• data_entitlement_query_oidc_issuer
• data_entitlement_query_oidc_client_id
• data_entitlement_query_oidc_audience
• data_entitlement_query_pe_rbac_service
• data_entitlement_query_pe_rbac_role_id
• data_entitlement_query_pe_rbac_ca_cert_file
• data_entitlement_ui_http_port
• data_entitlement_ui_https_port
• data_entitlement_manage_es
• data_entitlement_es_host
• data_entitlement_es_username
• data_entitlement_es_password
• data_entitlement_manage_s3
• data_entitlement_s3_endpoint
• data_entitlement_s3_region
• data_entitlement_s3_access_key
• data_entitlement_s3_secret_key
• data_entitlement_s3_facts_bucket
• data_entitlement_s3_force_path_style
• data_entitlement_s3_disable_ssl
• data_entitlement_user
• compose_version
• image_repository
• elasticsearch_image
• redis_image
• minio_image
• image_prefix
• ca_server
• ca_cert_file
• key_file
• cert_file
• allow_trust_on_first_use
• ui_use_tls
• ui_cert_files_puppet_managed
• ui_key_file
• ui_cert_file
• ui_ca_cert_file
• dns_name
• dns_alt_names
• version
• data_entitlement_version
• ui_version
• frontend_version
• docker_users
• max_es_memory
• prometheus_namespace
• access_log_level
• dashboard_url
• extra_hosts

create_docker_group

Data type: Boolean

Ensure the docker group is present.

Default value: true

manage_docker

Data type: Boolean

Install and manage docker as part of app_stack

Default value: true

log_driver

Data type: String[1]

The log driver Docker will use

Default value: 'journald'

data_dir

Data type: Optional[String[1]]

The data-root that docker will use to store volumes

Default value: undef

data_entitlement_port

Data type: Integer

Port to access HDP upload service

Default value: 9091

data_entitlement_query_port

Data type: Integer

Port to access HDP query service

Default value: 9092

data_entitlement_query_auth

Data type: Enum['basic_auth', 'oidc', 'pe_rbac', 'none']

What format to use for query authentication 'basic_auth' will use data_entitlement_query_username and data_entitlement_query_password to handle auth. 'oidc' will use data_entitlement_query_oidc_issuer and data_entitlement_query_oidc_client_id to handle auth. 'oidc' currently only supports Okta as an authn provider. 'pe_rbac' will cause the HDP to call out to a PE RBAC server to validate tokens from the UI. 'pe_rbac' is currently not supported by the UI, so only use this if you plan on making queries via the API directly. 'none' uses no auth for queries Defaults to 'none'

Default value: 'none'

data_entitlement_query_username

Data type: Optional[String[1]]

Username to add basic auth to query service

Default value: undef

data_entitlement_query_password

Data type: Optional[Sensitive[String[1]]]

Password to add basic auth to query service Can be a password string, but if it starts with a $, will be validated using Linux standards - $$$. Only algos of sha256 and sha512 are valid - $5$ and $6$. All other passwords will always be rejected.

Default value: undef

data_entitlement_query_oidc_issuer

Data type: Optional[String[1]]

The OIDC issuer. Currently only Okta URLs are supported.

Default value: undef

data_entitlement_query_oidc_client_id

Data type: Optional[String]

The client ID of the app in the OIDC issuer

Default value: undef

data_entitlement_query_oidc_audience

Data type: Optional[String]

The audience of the issued OIDC token

Default value: undef

data_entitlement_query_pe_rbac_service

Data type: Optional[Stdlib::HTTPUrl]

The URL of the pe_rbac_service Includes protocol, hostname, port, and rbac-api prefix if present, but no version number. Example: https://puppet:4433/rbac-api Required if data_entitlement_query_auth = 'pe_rbac'

Default value: undef

data_entitlement_query_pe_rbac_role_id

Data type: Integer

The role_id of the PE RBAC role allowed to query the HDP. Defaults to 1, which is the Administrator group by default. Superusers are allowed allowed to query the HDP.

Default value: 1

data_entitlement_query_pe_rbac_ca_cert_file

Data type: String

The CA Certfile to use for authenticate the RBAC Server Defaults to /etc/puppetlabs/puppet/ssl/certs/ca.pem Set to '-' to use system CAs

Default value: '/etc/puppetlabs/puppet/ssl/certs/ca.pem'

data_entitlement_ui_http_port

Data type: Integer

Port to access HDP UI via http

Default value: 80

data_entitlement_ui_https_port

Data type: Integer

Port to access HDP UI via https if ui_use_tls is true

Default value: 443

data_entitlement_manage_es

Data type: Boolean

= true Allow this module to manage elasticsearch If true, all other es parameters are ignored

Default value: true

data_entitlement_es_host

Data type: String[1]

Elasticsearch host to use

Default value: 'http://elasticsearch:9200/'

data_entitlement_es_username

Data type: Optional[String[1]]

Username to use to connect to elasticsearch

Default value: undef

data_entitlement_es_password

Data type: Optional[Sensitive[String[1]]]

Password to use to connect to elasticsearch

Default value: undef

data_entitlement_manage_s3

Data type: Boolean

Allow this module to manage S3 itself. If true, All other s3 parameters are ignored.

Default value: true

data_entitlement_s3_endpoint

Data type: String[1]

The S3 Endpoint to use

Default value: 'http://minio:9000/'

data_entitlement_s3_region

Data type: String[1]

The S3 Region to use

Default value: 'data_entitlement'

data_entitlement_s3_access_key

Data type: String[1]

The S3 Access Key to use

Default value: 'puppet'

data_entitlement_s3_secret_key

Data type: Sensitive[String[1]]

The S3 Secret Key to use

Default value: Sensitive('puppetpuppet')

data_entitlement_s3_facts_bucket

Data type: String[1]

The S3 Bucket to use for facts

Default value: 'facts'

data_entitlement_s3_force_path_style

Data type: Boolean

Disable AWS specific S3 Path Style

Default value: true

data_entitlement_s3_disable_ssl

Data type: Boolean

Disable SSL for the S3 backend

Default value: true

data_entitlement_user

Data type: String[1]

User to run HDP + all infra services as. Also owns mounted volumes Set to Puppet if certname == dns_name

Default value: '11223'

compose_version

Data type: String[1]

The version of docker-compose to install

Default value: '1.25.0'

image_repository

Data type: Optional[String[1]]

Image repository to pull images from - defaults to dockerhub. Can be used for airgapped environments/testing environments

Default value: undef

elasticsearch_image

Data type: String[1]

Elasticsearch image to use. Defaults to: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.1

Default value: 'docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.1'

redis_image

Data type: String[1]

Redis image to use. Defaults to: redis:6.2.4-buster

Default value: 'redis:6.2.4-buster'

minio_image

Data type: String[1]

Minio image to use. Defaults to: minio/minio:RELEASE.2021-04-22T15-44-28Z

Default value: 'minio/minio:RELEASE.2021-04-22T15-44-28Z'

image_prefix

Data type: String

Prefix that comes before each image Can be used for easy name spacing under the same repository

Default value: 'puppet/hdp-'

ca_server

Data type: Optional[String[1]]

URL of Puppet CA Server. If no keys/certs are provided, then HDP will attempt to provision its own certs and get them signed. Either this or ca_cert_file/key_file/cert_file can be specified. If autosign is not enabled, HDP will wait for the certificate to be signed by a puppet administrator

Default value: undef

ca_cert_file

Data type: Optional[String[1]]

CA certificate to validate connecting clients This or ca_server can be specified

Default value: undef

key_file

Data type: Optional[String[1]]

Private key for cert_file - pem encoded. This or ca_server can be specified

Default value: undef

cert_file

Data type: Optional[String[1]]

Puppet PKI cert file - pem encoded. This or ca_server can be specified

Default value: undef

allow_trust_on_first_use

Data type: Boolean

If true, then the HDP will download the CA and setup its certs and keys from the server if they haven't been provided. Since the HDP doesn't have any CA before starting this process, it will automatically trust the first server that appears as ca_server. If you don't explicitly set this value to true, then this module will not deploy an HDP installation without being given ca_cert_file, key_file, and cert_file.

Default value: true

ui_use_tls

Data type: Boolean

Use TLS for the UI and HDP Query endpoints

Default value: true

ui_cert_files_puppet_managed

Data type: Boolean

Indicate if the cert files used by the UI are managed by Puppet. If they are then a relationship is created between these files and the docker_compose resource so that containers are restarted when the contents of the files change, such as when the certificate is renewed.

Default value: false

ui_key_file

Data type: Optional[String[1]]

Key file to use for UI - pem encoded. Your browser should trust this you set ui_use_tls

Default value: undef

ui_cert_file

Data type: Optional[String[1]]

Cert file to use for UI - pem encoded. Your browser should trust this you set ui_use_tls

Default value: undef

ui_ca_cert_file

Data type: Optional[String[1]]

CA Cert file to use for UI - pem encoded. Setting this to anything but undef will cause the HDP to validate clients with mTLS If you don't have access to a puppet cert and key in your browser, do not set this parameter. It is unlikely that you want this value set.

Default value: undef

dns_name

Data type: String[1]

Name that puppet server will find HDP at. Should match the names in cert_file if provided. If ca_server is used instead, this name will be used as certname.

dns_alt_names

Data type: Array[String[1]]

Extra dns names attached to the puppet cert, can be used to bypass certname collisions

Default value: []

version

Data type: Optional[String[1]]

The super version to use of the components. If this is set, sets all the sub-component versions to this.

Default value: undef

data_entitlement_version

Data type: String[1]

The version of the HDP Data container to use

Default value: '0.0.1'

ui_version

Data type: Optional[String[1]]

The version of the HDP UI container to use If undef, defaults to data_entitlement_version

Default value: undef

frontend_version

Data type: Optional[String[1]]

The version of the HDP UI TLS Frontend container to use If undef, defaults to data_entitlement_version

Default value: undef

docker_users

Data type: Optional[Array[String[1]]]

Users to be added to the docker group on the system

Default value: undef

max_es_memory

Data type: String[1]

Max memory for ES to use - in JVM -Xmx{$max_es_memory} format. Example: 4G, 1024M. Defaults to 4G.

Default value: '4G'

prometheus_namespace

Data type: String[1]

The HDP data service exposes some internal prometheus metrics. This variable can be used to change the HDP's prom metric namespace.

Default value: 'data_entitlement'

access_log_level

Data type: Enum['none', 'all', 'admin']

Specify which level of access log the HDP should use. Defaults to 'admin', where the HDP logs each administrative action taken and the user behind it. The option 'all' causes all data queries to be logged.

Default value: 'admin'

dashboard_url

Data type: Optional[Stdlib::HTTPUrl]

The URL of the dashboard, where you would visit the UI in your browser. This parameter is used to configure alerts, so if you plan on using Relay or another alert handler, you should set this option if you want your links in your alerts to point you to the right place.

Default value: undef

extra_hosts

Data type: Hash[String[1], String[1]]

This parameter can be used to set hostname mappings in docker-compose file. Can be used to mimic the /etc/hosts techniques commonly used in puppet.

Default value: {}

data_entitlement::data_processor

Simple class to enable the HDP data processor

Examples

Configuration in a manifest with default port

# Settings applied to both a primary and compilers
class { 'profile::primary_and_compilers':
  class { 'data_entitlement::data_processor':
    data_entitlement_url => 'https://data_entitlement.example.com:9091',
  }
}

Configuration in a manifest with two HDP instances

# Settings applied to both a primary and compilers
class { 'profile::primary_and_compilers':
  class { 'data_entitlement::data_processor':
    data_entitlement_url =>
      'https://data_entitlement-prod.example.com:9091',
      'https://data_entitlement-staging.example.com:9091',
    ],
  }
}

Configuration in a manifest using PupeptDB instead of a facts terminus

# Settings applied to both a primary and compilers
class { 'profile::primary_and_compilers':
  class { 'data_entitlement::data_processor':
    data_entitlement_url           => 'https://data_entitlement.example.com:9091',
    collection_method => 'pdb_submit_only_server_urls',
  }
}

Configuration in a manifest using PupeptDB and an additional submit_only_server

# Settings applied to both a primary and compilers
class { 'profile::primary_and_compilers':
  class { 'data_entitlement::data_processor':
    data_entitlement_url                     => 'https://data_entitlement.example.com:9091',
    collection_method           => 'pdb_submit_only_server_urls',
    pdb_submit_only_server_urls => [
      'https://additional-destination.example.com',
    ],
  }
}

Configuration via Hiera with default port

---
data_entitlement::data_processor::data_entitlement_url: 'https://data_entitlement.example.com:9091'

Configuration via Hiera sending data to two HDP servers

---
data_entitlement::data_processor::data_entitlement_url:
  - 'https://data_entitlement-prod.example.com:9091'
  - 'https://data_entitlement-staging.example.com:9091'

Parameters

The following parameters are available in the data_entitlement::data_processor class:

• data_entitlement_url
• enable_reports
• manage_routes
• manage_pdb_submit_only_server_urls
• collection_method
• facts_terminus
• facts_cache_terminus
• collect_resources
• keep_node_re
• reports
• pdb_submit_only_server_urls

data_entitlement_url

Data type: Data_entitlement::Url

The url to send data to.

enable_reports

Data type: Boolean

Enable sending reports to HDP

Default value: true

manage_routes

Data type: Boolean

Enable managing the HDP routes file

Default value: true

manage_pdb_submit_only_server_urls

Data type: Boolean

This setting will allow the submit_only_server_urls setting in puppet.com to be set when $collection_method = 'pdb_submit_only_server_urls' and will allow the setting to be removed when $collection_method = 'facts_terminus'.

Default value: true

collection_method

Data type: Enum['facts_terminus', 'pdb_submit_only_server_urls']

Determine how the HDP will get its data. When set to facts_terminus, this module will setup a new facts terminus to send data. This is the preferred method. When set to pdb_submit_only_server_urls, this module will utlize the submit_only_server_urls setting of PuppetDB to have it send data as if the HDP server was another instance of PuppetDB. This method should only be used when the facts terminus method cannot be.

Default value: 'pdb_submit_only_server_urls'

facts_terminus

Data type: String[1]

Default value: 'data_entitlement'

facts_cache_terminus

Data type: String[1]

Default value: 'data_entitlement'

collect_resources

Data type: Boolean

Default value: true

keep_node_re

Data type: String[1]

Default value: '.*'

reports

Data type: String[1]

A string containg the list of report processors to enable

Default value: 'puppetdb,data_entitlement'

pdb_submit_only_server_urls

Data type: Optional[Array[Stdlib::HTTPSUrl]]

An array of PuppetDB instance URLs, including port number, to which commands should be sent, but which shouldn’t ever be queried for data needed during a Puppet run. This setting will use the value of $data_entitlement_url unless another value is provided.

Default value: undef

data_entitlement::proxy

This class takes care of configuring a node to run an HDP Proxy/Gateway.

Examples

Configure via Hiera

include data_entitlement::proxy

Parameters

The following parameters are available in the data_entitlement::proxy class:

• port
• user
• ca_server
• allow_trust_on_first_use
• ssl_dir
• ca_cert_file
• key_file
• cert_file
• dns_name
• dns_alt_names
• version
• prometheus_namespace
• token
• data_entitlement_address
• package
• service
• service_status
• service_enabled

port

Data type: Integer

Port to run HDP upload service on Defaults to 9091

Default value: 9091

user

Data type: String[1]

User to run HDP proxy as. Defaults to hdp-proxy

Default value: 'hdp-proxy'

ca_server

Data type: String[1]

URL of Puppet CA Server. If no keys/certs are provided, then HDP will attempt to provision its own certs and get them signed. Either this or ca_cert_file/key_file/cert_file can be specified. If autosign is not enabled, HDP will wait for the certificate to be signed by a puppet administrator

Default value: 'puppet'

allow_trust_on_first_use

Data type: Boolean

If true, then the HDP will download the CA and setup its certs and keys from the server if they haven't been provided. Since the HDP doesn't have any CA before starting this process, it will automatically trust the first server that appears as ca_server. If you don't explicitly set this value to true, then this module will not deploy an HDP installation without being given ca_cert_file, key_file, and cert_file.

Default value: false

ssl_dir

Data type: String[1]

The ssl dir for certificates Defaults to /etc/puppetlabs/hdp-proxy/ssl

Default value: '/etc/puppetlabs/hdp-proxy/ssl'

ca_cert_file

Data type: String[1]

CA certificate to validate connecting clients This or ca_server can be specified

Default value: "${data_entitlement::proxy::ssl_dir}/ca.cert.pem"

key_file

Data type: String[1]

Private key for cert_file - pem encoded. This or ca_server can be specified

Default value: "${data_entitlement::proxy::ssl_dir}/data-ingestion.key.pem"

cert_file

Data type: String[1]

Puppet PKI cert file - pem encoded. This or ca_server can be specified

Default value: "${data_entitlement::proxy::ssl_dir}/data-ingestion.cert.pem"

dns_name

Data type: String[1]

Name that puppet server will find HDP at. Should match the names in cert_file if provided. If ca_server is used instead, this name will be used as certname.

Default value: 'hdp-proxy'

dns_alt_names

Data type: Array[String[1]]

Extra dns names attached to the puppet cert, can be used to bypass certname collisions

Default value: []

version

Data type: String[1]

The version to use of the HDP Proxy. Defaults to latest

Default value: 'latest'

prometheus_namespace

Data type: String[1]

The HDP data service exposes some internal prometheus metrics. This variable can be used to change the HDP's prom metric namespace.

Default value: 'hdp_proxy'

token

Data type: Sensitive[String[1]]

The HDP's access token. Gathered from the the HDP UI when creating this proxy.

data_entitlement_address

Data type: Stdlib::HTTPUrl

The URL of the HDP endpoint to send data to.

Default value: 'https://hdp-upload-staging.prod.paas.puppet.net'

package

Data type: String[1]

The HDP proxy package name Defaults to hdp-proxy

Default value: 'hdp-proxy'

service

Data type: String[1]

The name of the HDP proxy service. Defaults to hdp-proxy

Default value: 'hdp-proxy'

service_status

Data type: String[1]

Indicates whether the HDP proxy service should be running or stopped Defaults to running

Default value: 'running'

service_enabled

Data type: Boolean

Indicates whether the HDP proxy service should be enabled Defaults to true

Default value: true

data_entitlement::resource_collector

This class adds module to the node, which adds our custom facts. Without it, we don't have the ability to pull all ral resources

Data types

Data_entitlement::Url

Data_entitlement::Url is a metatype that supports both single and multiple urls

Alias of

Variant[Array[Stdlib::HTTPUrl], Stdlib::HTTPUrl]

Tasks

get_service_status

Get the status of the HDP docker-compose services

Supports noop? false

update_all_images

Automates the dance needed to get docker-compose to update the container for every service

Supports noop? false

use_updated_image

Automates the dance needed to get docker-compose to use an updated image

Supports noop? false

Parameters

service

Data type: Enum[frontdoor,identity,influxdb,ingest-queue,mongo,query,rabbitmq,ui]

The service you wish to update

Change log

All notable changes to this project will be documented in this file. The format is based on Keep a Changelog and this project adheres to Semantic Versioning.

0.2.0 (2022-07-12)

Full Changelog

Added

• Update stdlib and inifile dependencies #5 (austb)
• (SDP-56) Replace proxy docker install with package #3 (austb)

* This Changelog was automatically generated by github_changelog_generator