pe_status_check
Version information
This version is compatible with:
- Puppet Enterprise 2023.8.x, 2023.7.x, 2023.6.x, 2023.5.x, 2023.4.x, 2023.3.x, 2023.2.x, 2023.1.x, 2023.0.x, 2021.7.x
- Puppet >= 7.18.0 < 9.0.0
- , , , , , , , ,
Plans:
- agent_state_summary
- agent_summary
- infra_role_summary
- infra_summary
Start using this module
Add this module to your Puppetfile:
mod 'puppetlabs-pe_status_check', '4.4.0'Learn more about managing modules with a PuppetfileDocumentation
puppetlabs-pe_status_check
- puppetlabs-pe_status_check
- Supporting Content
Description
puppetlabs-pe_status_check provides a way to alert the end-user when Puppet Enterprise is not in an ideal state. It uses pre-set indicators and has a simplified output that directs the end-user to the next steps for resolution.
Users of the tool have a greater ability to provide their own self-service resolutions and shorter incident resolution times with Puppet Support due to higher quality information available to our team.
Setup
What pe_status_check affects
This module installs two structured facts named pe_status_check and agent_status_check. Each fact contains an array of key pairs that output an indicator ID and a Boolean value. The pe_status_check fact is confined to only Puppet Enterprise infrastructure agents, and the agent_status_check fact is confined to non-infrastructure agent nodes.
Setup requirements
Install the module, plug-in sync will be used to deliver the required facts for this module, to each agent node in the environment the module is installed in.
Beginning with pe_status_check
This module primarily provides indicators using facts, so installing the module and allowing plug-in sync to occur lets the module start functioning.
Collection of the agent_status_check fact is disabled by default so as not to affect all puppet agents indiscriminately
Usage
The facts in this module can be directly consumed by monitoring tools such as Splunk, any element in the structured facts pe_status_check or agent_status_check reporting as false indicates a fault state in Puppet Enterprise. When any element reports as false, look up the incident ID in the reference section for next steps.
Alternatively, assigning the class pe_status_check to the infrastructure notifies on each Puppet run if any indicator is reporting as false, this can be viewed in the Puppet report for each node.
Enabling agent_status_check
By default your normal agent population will not collect the agent_status_check fact, this can be enabled for all agents or a subset of agents, by classifying pe_status_check::agent_status_enable to your nodes.
Disabling agent_status_check
Following the addition of the class pe_status_check::agent_status_enable to an agent node, disable the collection of agent_status_check fact, set the following parameter:
pe_status_check::agent_status_enable::agent_status_enabled = false
Reporting Options
Class declaration pe_status_check (optional)
To activate the notification functions of this module, classify your Puppet Infrastructure with the pe_status_check class using your preferred classification method. Below is an example using site.pp.
node 'node.example.com' {
include pe_status_check
}
For maximum coverage, report on all default indicators. However, if you need to make exceptions for your environment, classify the array parameter indicator_exclusions with a list of all the indicators you do not want to report on.
This workflow is not available for the agent_status_check fact.
class { 'pe_status_check':
indicator_exclusions => ['S0001','S0003','S0003','S0004'],
}
Ad-hoc Report (Plans)
The plans, pe_status_check::infra_summary and pe_status_check::agent_summary summarize the status of each of the checks on target nodes that have the pe_status_check or agent_status_check fact respectively, sample output can be seen below:
{
"nodes": {
"details": {
"pe-psql-70aefa-0.region-a.domain.com": {
"failed_tests_count": 0,
"passing_tests_count": 13,
"failed_tests_details": []
},
"pe-server-70aefa-0.region-a.domain.com": {
"failed_tests_count": 1,
"passing_tests_count": 30,
"failed_tests_details": [
"S0022 Determines if there is a valid Puppet Enterprise license in place at /etc/puppetlabs/license.key on your primary which is not going to expire in the next 90 days"
]
},
"pe-compiler-70aefa-0.region-a.domain.com": {
"failed_tests_count": 0,
"passing_tests_count": 23,
"failed_tests_details": []
},
"pe-compiler-70aefa-1.region-b.domain.com": {
"failed_tests_count": 0,
"passing_tests_count": 23,
"failed_tests_details": []
}
},
"failing": [
"pe-server-70aefa-0.region-a.domain.com"
],
"passing": [
"pe-compiler-70aefa-1.region-b.domain.com",
"pe-compiler-70aefa-0.region-a.domain.com",
"pe-psql-70aefa-0.region-a.domain.com"
]
},
"errors": {},
"status": "failing",
"failing_node_count": 1,
"passing_node_count": 3
}
The plan pe_status_check::infra_role_summary will provide you a hash with all PE infrastructure nodes, grouped by their role:
{
"primary": [
"primary.bastelfreak.local"
],
"replica": [
"replica.bastelfreak.local"
],
"compiler": [
"compiler01.bastelfreak.local",
"compiler02.bastelfreak.local"
],
"postgres": [],
"legacy_primary": [],
"legacy_compiler": []
}
The data is obtained from PuppetDB by checking the classes in the last catalog of every node. You can reuse the data in other plans or use it to inspect your environment. You can plott it in a more human-readable way with the puppet/format modules.
The plan pe_status_check::agent_state_summary provides you a hash with all nodes, grouped by failure state:
{
"noop": [ ],
"corrective_changes": [ ],
"used_cached_catalog": [ ],
"failed": [ ],
"changed": [ "student2.local" ],
"unresponsive": [ "student3.local", "student4.local", "student1.local", "login.local" ],
"responsive": [ "pe.bastelfreak.local"],
"unhealthy": [ "student2.local", "student3.local", "student4.local", "student1.local", "login.local" ],
"unhealthy_counter": 5,
"healthy": [ "pe.bastelfreak.local" ],
"healthy_counter": 1,
"total_counter": 6
}
noop: last catalog was applied in noop modefailed: The last catalog couldn't be compiled or catalog application raised an errorchanged: A node reported a changeunresponsive: Last report is older than 30 minutes (can be configured via therunintervalparameter)corrective_changes: A node reported corrective changesused_cached_catalog: The node didn't apply a new catalog but used a cached versionunhealthy: List of nodes that are in any of the above categoriesresponsive: Last report isn't older than 30 minutes (can be configured via therunintervalparameter). Doesn't matter if the report is healthy.healthy: All nodes - unhealthyunhealthy_counter: Amount of unhealthy nodeshealthy_counter: Amount of healthy nodestotal_counter: Amount of all nodes in PuppetDB
The goal of this plan is to run it before doing major upgrades, to ensure that your agents are in a healthy state.
To turn this into a table:
$result = run_plan('pe_status_check::agent_state_summary')
$table = format::table(
{
title => 'Puppet Agent states',
head => ['status check', 'Nodes'],
rows => $result.map |$key, $data| { [$key, [$data].flatten.join(', ')]},
}
)
out::message($table)
example output:
+------------------------------------------------+
| Puppet Agent states |
+---------------------+--------------------------+
| status check | Nodes |
+---------------------+--------------------------+
| noop | |
| corrective_changes | |
| used_cached_catalog | |
| failed | |
| changed | |
| unresponsive | |
| responsive | puppet.bastelfreak.local |
| unhealthy | |
| unhealthy_counter | 0 |
| healthy | puppet.bastelfreak.local |
| healthy_counter | 1 |
| total_counter | 1 |
+---------------------+--------------------------+
The plan has two parameters. By default it will log all unhealthy nodes. You can disable it by setting log_unhealthy_nodes to false. Then you will get:
{
"total_counter": 1,
"healthy_counter": 1,
"unhealthy_counter": 0
}
You can also enable the logging of of healthy nodes by setting log_healthy_nodes to true. In combination with log_unhealthy_nodes to false you get:
{
"healthy": [
"puppet.bastelfreak.local"
],
"total_counter": 1,
"healthy_counter": 1,
"unhealthy_counter": 0
}
Using a Puppet Query to report status.
As the pe_status_check module uses Puppet's existing fact behavior to gather the status data from each of the agents, it is possible to use PQL (puppet query language) to gather this information.
Consult with your local Puppet administrator to construct a query suited to your organizational needs. Please find some examples of using pe_client_tools to query the status check facts below:
-
To find the complete output of pe_status_check from all nodes listed by certname:
puppet query 'facts[certname,value] { name = "pe_status_check" }' -
To find the complete output of agen_status_check from all nodes listed by certname (this could be a large query based on the number of agent nodes, further filtering is advised ):
puppet query 'facts[certname,value] { name = "agent_status_check" }' -
To find those nodes with a specific status check set to false:
puppet query 'inventory[certname] { facts.pe_status_check.S0001 = false }'
Setup Requirements
pe_status_check::infra_summary and pe_status_check::agent_summary utilize hiera to lookup test definitions, this requires placing a static hierarchy in your environment level hiera.yaml.
plan_hierarchy:
- name: "Static data"
path: "static.yaml"
data_hash: yaml_data
See the following documentation for further explanation.
Using Static Hiera data to populate indicator_exclusions when executing plans
Place the plan_hierarchy listed in the step above, in the environment layer (https://www.puppet.com/docs/pe/latest/writing_plans_in_puppet_language_pe.html#using_hiera_with_plans)
Create a [static.yaml] file in the environment layer hiera data directory```
pe_status_check::indicator_exclusions:
- '<TEST ID>'
Indicator ID's within array will be excluded when running pe_status_check::infra_summary and pe_status_check::agent_summary p
Running the plans
The pe_status_check::infra_summary and pe_status_check::agent_summary plans can be run from the PE console or from the command line. Below are some examples of running the plans from the command line. More information on the parameters in the plan can be seen in the REFERENCE.md.
Example call from the command line to run pe_status_check::infra_summary against all infrastructure nodes:
puppet plan run pe_status_check::infra_summary
Example call from the command line to run pe_status_check::agent_summary against all regular agent nodes:
puppet plan run pe_status_check::agent_summary
Example call from the command line to run against a set of infrastructure nodes:
puppet plan run pe_status_check::infra_summary targets=pe-server-70aefa-0.region-a.domain.com,pe-psql-70aefa-0.region-a.domain.com
Example call from the command line to exclude indicators for pe_status_check::infra_summary:
puppet plan run pe_status_check::infra_summary -p '{"indicator_exclusions": ["S0001","S0021"]}'
Example call from the command line to exclude indicators for pe_status_check::agent_summary:
puppet plan run pe_status_check::agent_summary -p '{"indicator_exclusions": ["AS001","AS002"]}'
Reference
Fact: pe_status_check_role
This fact is used to determine which individual status checks should be run on each individual infrastructure node. The fact queries which Puppet Enterprise Roles have been classified to each node and uses this to make the determination.
| Role | Description |
|---|---|
| primary | The node is both a certificate authority and a postgres host |
| replica | The node has the primary_master_replica role |
| pe_compiler | The node has both the master and puppetdb roles |
| postgres | The node has just the database role |
| legacy_compiler | The node has the master role but not the puppetdb role |
| legacy_primary | The node is a certificate authority but not a postgres host |
| unknown | The node type could not be determined |
A failure to determine node type will result in a safe subset of checks being run that will work on all infrastructure node types.
Fact: pe_status_check
This fact is confined to run on infrastructure nodes only.
Refer below for next steps when any indicator reports a false.
| Indicator ID | Description | Self-service steps | What to include in a Support ticket |
|---|---|---|---|
| S0001 | Determines if the puppet service is running on agents. | See documentation | If the service fails to start, open a Support ticket referencing S0001, and provide syslog and any errors output when attempting to restart the service. |
| S0002 | Determines if the pxp-agent service is running. | Start the pxp-agent service - puppet resource service pxp-agent ensure=running, if the service has failed check the logs located in /var/logs/puppetlabs/pxp-agent, for information to debug and understand what the logs mean, see the following links for assistance. Connection Type Issues, if the service is up and running but issues still occur, see Debug Logging |
If the service fails to start, open a Support ticket referencing S0002, provide syslog any errors output when attempting to restart the service, and /var/log/puppetlabs/pxp-agent/pxp-agent.log |
| S0003 | Determines if infrastructure components are running in noop. | Do not routinely configure noop on PE infrastructure nodes, as it prevents the management of key infrastructure settings. Disable this setting on infrastructure components. | If you are unable to disable noop or encounter an error when disabling noop, open a Support ticket referencing S0003, and provide any errors output when attempting to change the setting. |
| S0004 | Determines if the Puppet Server status endpoint is returning any errors. | Execute puppet infrastructure status. Which ever service returns in a state that is not running, examine the logging for that service to indicate the fault. |
Open a Support ticket referencing S0004, provide the output of puppet infrastructure status and any service logs associated with the errors. |
| S0005 | Determines if certificate authority (CA) cert expires in the next 90 days. | Install the puppetlabs-ca_extend module and follow steps to extend the CA cert. | Open a Support ticket referencing S0005 and provide support script output from the primary server, and any errors encountered when using the ca_extend module. |
| S0006 | Determines if Puppet metrics collector is enabled and collecting metrics. | Metrics collector is a tool that lets you monitor a PE installation. If it is not enabled, enable it. | If you have issues enabling metrics, open a ticket referencing S0006 and provide the output of the support script. |
| S0007 | Determines if there is at least 20% disk free on the PostgreSQL data partition. | Determines if growth is slow and expected within the TTL of your data. If there's an unexpected increase, use this article to troubleshoot PuppetDB | If your Puppet Practitioner is unable to find a cause for the growth and the suggested KB does not help, open a Support ticket referencing S0007 and provide details about large files and folders, rate of growth, and a full support script from the affected node. |
| S0008 | Determines if there is at least 20% disk free on the codedir data partition. | See documentation | |
| S0009 | Determines if pe-puppetserver service is running and enabled on relevant components. | See documentation | If you are unable to explain the service outage from the logging, or are unable to start the service, open a Support ticket referencing S0009 and provide output of the support script |
| S0010 | Determines if pe-puppetdb service is running and enabled on relevant components. | See documentation | If you are unable to explain the service outage from the logging, or are unable to start the service, open a Support ticket referencing S0010 and provide output of the support script |
| S0011 | Determines if pe-postgres service is running and enabled on relevant components. | See documentation | If you are unable to explain the service outage from the logging, or are unable to start the service, open a Support ticket referencing S0011 and provide output of the support script |
| S0012 | Determines if Puppet produced a report during the last run interval. | Troubleshoot Puppet run failures. | Open a Support ticket referencing S0012 and provide the output of puppet agent -td > debug.log 2>&1 |
| S0013 | Determines if the catalog was successfully applied during the last Puppet run. | Troubleshoot Puppet run failures. | Open a Support ticket referencing S0013 and provide the output of puppet agent -td > debug.log 2>&1 |
| S0014 | Determines if anything in the command queue is older than a Puppet run interval. | This can indicate that the PuppetDB performance is inadequate for incoming requests. Review PuppetDB performance. Use metrics to pinpoint the issue. | If your are unable to determine the reason from the metrics, open a Support ticket referencing S0014 and provide the output of the support script. and the findings from your analysis of the metrics |
| S0015 | Determines if the infrastructure agent host certificate is expiring in the next 90 days. | Puppet Enterprise has built in functionalilty to regenerate infrastructure certificates, see the following documentation | If the documented steps fail to resolve your issue, open a support ticket referencing S0015 and provide the error message received when running the steps. |
| S0016 | Determines if there are any OutOfMemory errors in the puppetserver log. |
Increase the Java heap size for that service. | Open a Support ticket referencing S0016 and provide puppet metrics, /var/log/puppetlabs/puppetserver/puppetserver.log, and the output of puppet infra tune. |
| S0017 | Determines if there are any OutOfMemory errors in the puppetdb log. |
Increase the Java heap size for that service. | Open a Support ticket referencing S0017 and provide puppet metrics, /var/log/puppetlabs/puppetdb/puppetdb.log, and the output of puppet infra tune. |
| S0018 | Determines if there are any OutOfMemory errors in the orchestrator log. |
Increase the Java heap size for that service. | Open a Support ticket referencing S0018 and provide puppet metrics, /var/log/puppetlabs/orchestration-services/orchestration-services.log, and output of puppet infra tune. |
| S0019 | Determines if there are sufficent jRubies available to serve agents. | Insufficient jRuby availability results in queued puppet agents and overall poor system performance. There can be many causes: Insufficient server tuning for load, a thundering herd, and insufficient system resources for scale. | If self-sevice fails to resolve the issue, open a ticket referencing S0019 and provide a description of actions so far and the output of the support script. |
| S0020 | Determines if the Console status api reports all services as running | Determine which service caused the failure Service Request Format, go to the [logging] (https://www.puppet.com/docs/pe/2023.4/what_gets_installed_and_where.html?_ga=2.219585753.1594518485.1698057844-280774152.1694007045&_gl=1*xeui3a*_ga*MjgwNzc0MTUyLjE2OTQwMDcwNDU.*_ga_7PSYLBBJPT*MTY5ODMyNzY5MS41Ny4xLjE2OTgzMjgyOTIuMTEuMC4w#log_files_installed) of that service and look for related error messages | Open a Support ticket referencing S0020, please provide the name of the service that failed, time of failure, error messages and provide a copy of the Support Script from your primary. |
| S0021 | Determines if free memory is less than 10%. | Ensure your system hardware availability matches the recommended configuration, note this assumes no third-party software using significant resources, adapt requirements accordingly for third-party requirements. Examine metrics from the server and determine if the memory issue is persistent | If you have issues with memory utilization in Puppet Enterprise that can not be explained, open a Support ticket, referencing S0021 and provide the output of the support script |
| S0022 | Determines the validity of both older and newer types of Puppet Enterprise licenses. | Get help with Puppet Enterprise license issues | Open a Support ticket referencing S0022 and provide the output of the following commands ls -la /etc/puppetlabs/license.key and cat /etc/puppetlabs/license.key. |
| S0023 | Determines if certificate authority CRL expires in the next 90 days. | The solution is to reissue a new CRL from the Puppet CA, note this will also remove any revoked certificates. To do this follow the instructions in this module | Open a Support ticket referencing S0023 and provide support script output from the primary server, and errors or output collected from the resolution steps |
| S0024 | Determines if there are files in the puppetdb discard directory newer than 1 week old | see documentation | If you are unable to determine a reason for the rejections from logging, Open a Support ticket referencing S0024 and provide a copy of the PuppetDB log for the time in question, along with a sample of the most recent file in the following directory /opt/puppetlabs/server/data/puppetdb/stockpile/discard/ |
| S0025 | Determines if the host copy of the CRL expires in the next 90 days. | If the Output of S0023 on the primary server is also false use the resolution steps in S0023. If S0023 on the Primary is True, follow this article | Open a Support ticket referencing S0025 and provide any errors you received in following the resolution steps |
| S0026 | Determines if pe-puppetserver JVM Heap-Memory is set to an inefficient value. | Due to an odditity in how JVM memory is utilised, most applications are unable to consume heap memory between ~31GB and ~48GB as such is if you have heap memory set within this value, you should reduce it to more efficiently allocate server resources. To set heap refer to Increase the Java heap size for this service. | |
| S0027 | Determines if if pe-puppetdb JVM Heap-Memory is set to an inefficient value. | Due to an odditity in how JVM memory is utilised, most applications are unable to consume heap memory between ~31GB and ~48GB as such is if you have heap memory set within this value, you should reduce it to more efficiently allocate server resources. To set heap refer to Increase the Java heap size for this service. | |
| S0029 | Determines if number of current connections to Postgresql DB is approaching 90% of the max_connections defined. |
See documentation | Should you be unable to determine the reason for a recent increase in connection use, or are having issue upping the number of connections available, open a Support ticket referencing S0029 and provide the current and future value for puppet_enterprise::profile::database::max_connections and we will assist. |
| S0030 | Determines when infrastructure components have the setting use_cached_catalog set to true. |
Don't configure use_cached_catalog on PE infrastructure nodes. It prevents the management of key infrastructure settings. Disable this setting on all infrastructure components. See our documentation for more information | If you encounter errors after disabling use_cached_catalog, open a Support ticket referencing S0030 and provide the errors. |
| S0031 | Determines if old PE agent packages exist on the primary server. | Remove the old PE agent packages. | |
| S0033 | Determines if Hiera 5 is in use. | Upgrading to Hiera 5 offers some major advantages | If you're having issues upgrading to Hiera 5 or if your global Hiera configuration file was erroneously modified, open a Support ticket referencing S0033. Provide your global Hiera configuration file puppet config print hiera_config; the default location is /etc/puppetlabs/puppet/hiera.yaml. |
| S0034 | Determines if your PE deployment has not been upgraded in the last year. | Upgrade your PE instance. | If you have issues during a Puppet Upgrade, open a ticket and provide your current version and the version you would like to upgrade to and state any problems, providing any logging that is helpful. |
| S0035 | Determines if puppet module list is returning any warnings |
If S0035 returns false, i.e., warnings are present, you should run puppet module list --debug and resolve the issues shown. The Puppetfile does NOT include Forge module dependency resolution. You must make sure that you have every module needed for all of your specified modules to run.Please refer to Managing environment content with a Puppetfile for more info on Puppetfile and refer to the specific module page on the forge for further information on specific dependancies |
If you are unable to remove all the warnings, then please refer to Get help for supported modules and raise a support request |
| S0036 | Determines if max-queued-requests is set above 150. |
The maximum value for jruby_puppet_max_queued_requests is 150 |
If you are unable to change the value of jruby_puppet_max_queued_requests or encounter an error when changing it, open a Support ticket referencing S0036 and provide any errors output when attempting to change the setting. |
| S0038 | Determines whether the number of environments within $codedir/environments is less than 100 |
Having a large number of code environments can negatively affect Puppet Server performance. See the Configuring Puppet Server documentation for more information. You should examine if you need them all, any unused environments should be removed. If all are required you can ignore this warning. | |
| S0039 | Determines if Puppets Server has reached its queue-limit-hit-rate,and is sending messages to agents. |
Check the max-queued-requests article for more information. | If the article is unable to solve your issue, open a Support ticket referencing S0039, indicating the investigation so far, and any issues you encountered, then provide the support script output from the primary server. |
| S0040 | Determines if PE is collecting system metrics. | If system metrics are not collected by default, the sysstat package is not installed on the impacted PE infrastructure component. Install the package and set the parameter puppet_enterprise::enable_system_metrics_collection to true. See the documentation. |
After system metrics are configured, you do not see any files in /var/log/sa or if the /var/log/sa directory does not exist, open a Support ticket. |
| S0041 | Determines if the pxp broker on a compiler has an established connection to another pxp broker | See documenation | If unable to make a connection to a broker, raise a ticket with the support team quoting S0041 and attaching the file /var/log/puppetlabs/puppetserver/pcp-broker.log along with the conclusions of your investigation so far |
| S0042 | Determines if the pxp-agent has an established connection to a pxp broker | See documenation | If unable to make a connection to a broker, raise a ticket with the support team quoting S0042 and attaching the file /var/log/puppetlabs/pxp-agent/pxp-agent.log (on *nix) or C:/ProgramData/PuppetLabs/pxp-agent/var/log/pxp-agent.log (on Windows), along with the conclusions of your investigation so far |
| S0043 | Determines if there are nodes with Puppet agent versions ahead of the primary server | Agent nodes should not be running Puppet agent versions ahead of infrastructure nodes. Instead consider upgrading PE so that PE package management contains the desired Puppet agent version. See the upgrading PE and upgrading agents documentation for more information. | If you are unable to determine why the indicator is evaluating to false or have questions about Puppet agent versions, open a support ticket and reference S0043. |
| S0044 | Determines if Puppet Servers are using the the PE classifier for the node data plugin (node terminus) | Due to performance optimizations, it is recommended to use the PE classifier plugin instead of external node classifier (ENC) scripts or applications. See the node_terminus configuration setting documentation for more information. | If you have additional questions about the node_terminus configuration setting, open a support ticket and reference S0044. |
| S0045 | Determines if Puppet Servers are configured with an excessive number of JRubies. | Because each JRuby instance consumes additional memory, having too many can reduce the amount of heap space available to Puppet server and cause excessive garbage collections. While it is possible to increase the heap along with the number of JRubies, we have observered diminishing returns with more than 12 JRubies and therefore recommend an upper limit of 12. We also recommend allocating between 1 - 2gb of heap memory for each JRuby. | If you would like to measure the effects of changing JRubies and heap settings, use the Puppet Operational Dashboards module to configure a metrics stack and Grafana dashboards for viewing the metrics. If you still have performance issues or further questions, open a support ticket and reference S0045. |
Fact: agent_status_check
This fact is confined to run on only agent nodes that are NOT infrastructure nodes.
Refer below for next steps when any indicator reports a false.
| Indicator ID | Description | Self-service steps | What to include in a Support ticket |
|---|---|---|---|
| AS001 | Determines if the agent host certificate is expiring in the next 90 days. | Puppet Enterprise has a plan built into extend agent certificates. Use a puppet query to find expiring host certificates and pass the node ID to this plan: puppet plan run enterprise_tasks::agent_cert_regen agent=$(puppet query 'inventory[certname] { facts.agent_status_check.AS001 = false }' \| jq -r '.[].certname' \| paste -sd, -) master=$(puppet config print certname) |
If the plan fails to run, open a support ticket referencing AS001 and provide the error message received when running the plan. |
| AS002 | Determines if the pxp-agent has an established connection to a pxp broker | Ensure the pxp-agent service is running, if running check /var/log/puppetlabs/pxp-agent/pxp-agent.log (on *nix) or C:/ProgramData/PuppetLabs/pxp-agent/var/log/pxp-agent.log (on Windows) — Contains the for connection issues, first ensuring the agent is connecting to the proper endpoint, for example, a compiler and not the primary. This fact can also be used as a target filter for running tasks, ensuring time is not wasted sending instructions to agents not connected to a broker |
If unable to make a connection to a broker, raise a ticket with the support team quoting AS002 and attaching the file /var/log/puppetlabs/pxp-agent/pxp-agent.log (on *nix) or C:/ProgramData/PuppetLabs/pxp-agent/var/log/pxp-agent.log (on Windows) along with the conclusions of your investigation so far |
| AS003 | Determines the certname configuration parameter is incorrectly set outside of the [main] section of the puppet.conf file. | The Puppet documentation states clearly certname should always be placed solely in the [main] section to prevent unforseen issues with the operation of the puppet agent https://puppet.com/docs/puppet/7/configuration.html#certname | If unable to determine why the indicator is being raised. Open a ticket with the support team quoting AS003 and attaching the file puppet.conf along with the conclusions of your investigation so far . |
| AS004 | Determines if the host copy of the CRL expires in the next 90 days. | If the Output of S0023 on the primary server is also false use the resolution steps in S0023. If S0023 on the Primary is True, follow this article | Open a Support ticket referencing AS004 and provide any errors you recieved in following the resolution steps |
How to report an issue or contribute to the module
If you are a PE user and need support using this module or encounter issues, our Support team is happy to help you. Open a ticket at the Support Portal. If you have a reproducible bug or are a community user, you can open an issue directly in the GitHub issues page of the module. We also welcome PR contributions to improve the module. Please see further details about contributing.
Supporting Content
Articles
The Support Knowledge base is a searchable repository for technical information and how-to guides for all Puppet products.
This Module has the following specific Article(s) available:
Videos
The Support Video Playlist is a resource of content generated by the support team
This Module has the following specific video content available:
Reference
Table of Contents
Classes
pe_status_check: This class should be enabled if you wish Puppet to notify when pe_status_check indicators are not at optimal valuespe_status_check::agent_status_enable: Enables the execution of agent_status_check fact
Plans
pe_status_check::agent_state_summary: provides an overview of all Puppet agents and their error statespe_status_check::agent_summary: Summary report of the state of agent_status_check on each node Uses the facts task to get the current status from each node and produces a summary report in JSONpe_status_check::infra_role_summary: provides an overview of all PE systems and their rolepe_status_check::infra_summary: Summary report if the state of pe_status check on each node Uses the facts task to get the current status from each node and produces a summary report in JSON
Classes
pe_status_check
When this class is enabled, when any of the indicators in the pe_status_check fact are false puppet will notify of this, individual tests can be disabled by adding the ID to the indicator_exclusions parameter
Examples
include pe_status_check
Parameters
The following parameters are available in the pe_status_check class:
indicator_exclusions
Data type: Array[String[1]]
List of disabled indicators, place any indicator ids you do not wish to report on in this list
Default value: []
checks
Data type: Hash
Hash containing a descriptiong for each key indicator
pe_status_check::agent_status_enable
Adding this class will enable the execution of the agent_status_check fact, This allows the fact to be targeted to a specific agent or group of agents
Examples
include pe_status_check::agent_status_enable
Parameters
The following parameters are available in the pe_status_check::agent_status_enable class:
agent_status_enabled
Data type: Boolean
Flag to enable or disable agent_status_check fact
Default value: true
Plans
pe_status_check::agent_state_summary
provides an overview of all Puppet agents and their error states
Parameters
The following parameters are available in the pe_status_check::agent_state_summary plan:
runinterval
Data type: Integer[0]
the runinterval for the Puppet Agent in minutes. We consider latest reports that are older than runinterval as unresponsive
Default value: 30
log_healthy_nodes
Data type: Boolean
optionally return all healthy nodes, not only the unhealthy
Default value: false
log_unhealthy_nodes
Data type: Boolean
optionally hide unhealthy nodes
Default value: true
pe_status_check::agent_summary
Summary report of the state of agent_status_check on each node Uses the facts task to get the current status from each node and produces a summary report in JSON
Parameters
The following parameters are available in the pe_status_check::agent_summary plan:
targets
Data type: Optional[TargetSpec]
A comma seprated list of FQDN's of Puppet agent nodes Defaults to using a PuppetDB query to identify nodes
Default value: undef
indicator_exclusions
Data type: Array[String[1]]
List of disabled indicators, place any indicator ids you do not wish to report on in this list Static Hiera Data can be used to set indicator_exclusions in a plan - for more information see https://www.puppet.com/docs/pe/latest/writing_plans_in_puppet_language_pe.html#using_hiera_with_plans
Default value: lookup('pe_status_check::indicator_exclusions', undef, undef, [])
pe_status_check::infra_role_summary
provides an overview of all PE systems and their role
pe_status_check::infra_summary
Summary report if the state of pe_status check on each node Uses the facts task to get the current status from each node and produces a summary report in JSON
Parameters
The following parameters are available in the pe_status_check::infra_summary plan:
targets
Data type: Optional[TargetSpec]
A comma seprated list of FQDN's of Puppet infrastructure agent nodes Defaults to using a PuppetDB query to identify nodes
Default value: undef
indicator_exclusions
Data type: Array[String[1]]
List of disabled indicators, place any indicator ids you do not wish to report on in this list Static Hiera Data can be used to set indicator_exclusions in a plan - for more information see https://www.puppet.com/docs/pe/latest/writing_plans_in_puppet_language_pe.html#using_hiera_with_plans
Default value: lookup('pe_status_check::indicator_exclusions', undef, undef, [])
What are plans?
Modules can contain plans that take action outside of a desired state managed by Puppet. It’s perfect for troubleshooting or deploying one-off changes, distributing scripts to run across your infrastructure, or automating changes that need to happen in a particular order as part of an application deployment.
Change log
All notable changes to this project will be documented in this file. The format is based on Keep a Changelog and this project adheres to Semantic Versioning.
v4.4.0 (2024-09-19)
Added
- (SUP-4955) Wildcard Operator Added for file license ingestion #233 (Aaronoftheages)
- Add plan for Puppet agent state summary #226 (bastelfreak)
v4.3.0 (2024-07-01)
Added
- (SUP-4929) Update to S0022 for handling license files with .key and .lic file #230 (Aaronoftheages)
- Add infra_role_summary plan #225 (bastelfreak)
UNCATEGORIZED PRS; LABEL THEM ON GITHUB
v4.2.0 (2024-03-20)
Added
- (SUP-4724/SUP-4728/SUP-4729) added self resolution documentation links in README for S0024, S0041 and S0042 #222 (Aaronoftheages)
- (SUP-4722/SUP-4723/SUP-4727) added self resolution documentation links in README for S0010, S0011 and S0029 #221 (Aaronoftheages)
- (SUP-4721) added self resolution documentation link in README for S0009 #220 (Aaronoftheages)
- (SUP-4671) added self resolution documentation link in README for S0008 #218 (Aaronoftheages)
- (SUP-4786) ) added Self Resolution Documentation to S0002 with Formatting Fix #217 (Aaronoftheages)
v4.1.0 (2023-12-11)
Added
- (MAINT) Edit indicator descriptions for consistent style and accurate grammar #210 (J-Hunniford)
- (SUP-4625) Add check for excessive JRubies #209 (m0dular)
Fixed
- (SUP-4714) Check if logfile exists during runtime of S0039 #214 (Aaronoftheages)
v4.0.1 (2023-10-26)
Fixed
- (SUP-4585) Update Readme for missing description of S0020 #207 (Aaronoftheages)
v4.0.0 (2023-10-18)
Changed
Added
- (SUP-3709) - Indicator Exclusion using code manager and Hiera lookup #205 (Aaronoftheages)
- SUP-4458 addition of test S0020 to test console-service endpoint #204 (Aaronoftheages)
Fixed
- (SUP-4402) pe_status_check fails if hiera.yaml is empty #200 (MartyEwings)
v3.0.0 (2023-06-12)
Changed
- (SUP-3952) Remove Puppet 6 as a supported platform #186 (elainemccloskey)
Added
- (SUP-4275) Update Hiera definition of S0038 #191 (MartyEwings)
- (SUP-4192) Puppet 8 release prep #188 (elainemccloskey)
Fixed
- (SUP-4282) add error handling to S0044 #192 (taikaa)
- (SUP-4129) Empty Yaml load error handling and PDK Update #185 (MartyEwings)
- (SUP-4076) Make the check time period 2* run interval #184 (MartyEwings)
v2.6.0 (2023-01-23)
Added
- (SUP-3703) Add indicator S0044 node_terminus is the PE classifier #181 (taikaa)
- (SUP-3696) Added S0043 to determine whether there are nodes with Puppet agent versions ahead of the primary server #180 (taikaa)
v2.5.1 (2022-11-10)
Fixed
- (SUP-3786) Update s00036 to not trigger on 150 #178 (elainemccloskey)
v2.5.0 (2022-11-09)
Added
Fixed
- (SUP-3777) Stop loop in S0039 if log is in unreadable format #174 (elainemccloskey)
- (#170) Improve agent_status_check.AS002 performance #171 (seanmil)
v2.4.1 (2022-10-14)
Fixed
- (SUP-3724) error handling for access log parsing #167 (MartyEwings)
- (Sup-3700) Handle missing licence type #165 (MartyEwings)
- Remove duplicate Chunks #164 (MartyEwings)
v2.4.0 (2022-09-29)
Added
- (SUP-3665) add check for ineffeicent heap max values #161 (MartyEwings)
Fixed
- (sup-3676) Handle Nill Value in S0038 #162 (MartyEwings)
- (SUP-3622)Correct description of S0021 in hiera #160 (MartyEwings)
- (SUP-3594) readme updates for clarity #158 (MartyEwings)
v2.3.1 (2022-08-15)
Fixed
v2.3.0 (2022-07-27)
Added
- (SUP-3362) add CRL expiration check #149 (MartyEwings)
Fixed
v2.2.0 (2022-07-15)
Added
- (Sup-3465) check for certname in incorrect configuration section #147 (MartyEwings)
v2.1.1 (2022-07-11)
Fixed
- (SUP-3452) Handle hostcert path missing #145 (MartyEwings)
v2.1.0 (2022-07-08)
Added
- (SUP-3450) make agent_status_check collection optional #141 (MartyEwings)
- (SUP-3426) Use shorter timeout for facter execute #139 (m0dular)
- (SUP-3400)(SUP-3401)(SUP-3402) Update Readme to explain facts further #137 (MartyEwings)
Fixed
- SUP-3457 get_resource error handling #143 (m0dular)
- (SUP-3442) Update ss commands so timeout kills the process #142 (elainemccloskey)
v2.0.2 (2022-06-17)
Fixed
- SUP-3398 legacy primary support #135 (MartyEwings)
v2.0.1 (2022-06-14)
Fixed
v2.0.0 (2022-06-13)
Changed
v1.5.0 (2022-05-27)
Added
- (SUP-3285) Update readme for S0035 #129 (elainemccloskey)
v1.4.0 (2022-04-29)
Added
- (SUP-3230) Check for the expiry date of the host cert on infrastructure #123 (MartyEwings)
- (SUP-3229) Summary Plan for agents #122 (MartyEwings)
- (SUP-3005) Update plan output format #117 (jarretlavallee)
- Add Suse support to AS002 #115 (seanmil)
- (SUP-3005) Add Plan for Reporting on Infrastructure #114 (MartyEwings)
- (SUP-2914) Verify that 'puppet module list' output is without Warnings #43 (gavindidrichsen)
v1.3.0 (2022-04-07)
Added
- (SUP-3150) Broker TCP Checks for infra components #109 (MartyEwings)
- (SUP-3121) Agent connection to pxp broker #106 (MartyEwings)
- (SUP-2917) Add indicator S0038 to check number of environments that are present in $codedir/environments #105 (taikaa)
- (SUP-2908) check current connections to Postgres less than 90% defined maximum #104 (sandrajiang)
Fixed
- (SUP-3180) Rescue a loaderror when checking filesystem #111 (jarretlavallee)
- (SUP-3101) Add exception handling and Facter warnings for license_type and end date that do not exist or are invalid. Fact no longer resolves to true as a catchall. Change license_type and end_date variable assignments to first item of array rather than converting entire array to a string. Update spec test. #103 (taikaa)
- (SUP-3122) Fix PSQL node detection in 2021.5 #98 (MartyEwings)
v1.2.0 (2022-03-23)
Added
- (SUP-2903) Check for new items in discard Directory #92 (MartyEwings)
Fixed
- Check for non-nil match in S0039 #94 (m0dular)
- (SUP-3116) Fix replica Detection in 2021.5 #93 (MartyEwings)
- (SUP-3099) Only Alert on 503 messages sent in last run interval #91 (elainemccloskey)
v1.1.0 (2022-02-24)
Added
v1.0.0 (2022-02-08)
Changed
- Rename Module to pe_status_check #74 (MartyEwings)
Added
- Adds Details to Warning Messages #76 (MartyEwings)
- Improve documentation #69 (MartyEwings)
- (SUP-2945) Updating S0004 to call the services API #68 (jordi-garcia)
- (SUP-2892) - No recent OOM errors logged in any JVM #58 (BartoszBlizniak)
- (SUP-2890) - Check command queue depth #56 (BartoszBlizniak)
- (SUP-2901) License check, tests and readme update #51 (pgrant87)
- (SUP-2896) Check Avg Free JRubies #50 (coreymbe)
- (SUP-2910) Check if older packages are still available to pe_repo #47 (asselvakumar)
- (SUP-2918) Check If Puppet Server is sending 503s to agents #44 (kksubbu72)
- (SUP-2946) Check Metrics Collector is enabled #41 (MartyEwings)
- (SUP-2913) Puppet not updated for more than a year #38 (gmcgrillan)
- (SUP-2912) Added S0033 to check if Global Hiera is Hiera 5, added tests, updated README #35 (taikaa)
- Adding Agent only Fact check for expiring host cert #32 (MartyEwings)
- (SUP-2919) - is puppet_metrics_collector::system configured #29 (tcwest23)
- (SUP-2915) Check if max-queued-requests is configured to above 150 #28 (elainemccloskey)
- (SUP-2909) Check that use_cached_catalog setting is false #27 (elainemccloskey)
Fixed
- Add confinement to S0036 #36 (elainemccloskey)
v0.2.0 (2022-01-11)
Added
- (SUP-2898) Available Memory on the platform is less than 10% #18 (MartyEwings)
v0.1.0 (2021-12-14)
Added
- (SUP-2862) Basic Readme for version 0.1.0 #13 (MartyEwings)
- (SUP-2861) Puppet Agent Health Checks #12 (MartyEwings)
- Adds a filesystem free method #9 (jarretlavallee)
- Adds in PE node confinement types and more status checks #8 (MartyEwings)
- Adds additional indicators #5 (MartyEwings)
- (maint) Move service logic to a module #3 (jarretlavallee)
Fixed
- (fix) Do not filter on Undef #10 (jarretlavallee)
* This Changelog was automatically generated by github_changelog_generator
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Quality checks
We run a couple of automated scans to help you assess a module’s quality. Each module is given a score based on how well the author has formatted their code and documentation and select modules are also checked for malware using VirusTotal.
Please note, the information below is for guidance only and neither of these methods should be considered an endorsement by Puppet.
Malware scan results
The malware detection service on Puppet Forge is an automated process that identifies known malware in module releases before they’re published. It is not intended to replace your own virus scanning solution.
Learn more about malware scans- Module name:
- puppetlabs-pe_status_check
- Module version:
- 4.4.0
- Scan initiated:
- September 19th 2024, 3:19:02
- Detections:
- 0 / 63
- Scan stats:
- 63 undetected
- 0 harmless
- 0 failures
- 0 timeouts
- 0 malicious
- 0 suspicious
- 14 unsupported
- Scan report:
- View the detailed scan report