puppetlabs-patching_as_code

Table of Contents

• puppetlabs-patching_as_code

  - [Table of Contents](#table-of-contents)
  

  • Description
  • Setup
    • What this module affects
    • Setup Requirements
    • Beginning with patching_as_code
  • Usage
    • Customizing the patch groups
  • Controlling which patches get installed
  • Defining situations when patching needs to be skipped
  • Defining pre/post-patching and pre-reboot commands
  • Limitations

Description

This module provides automatic patch management for Linux and Windows through desired state code.

Setup

What this module affects

This module will leverage the fact data provided by either the albatrossflavour/os_patching or PE 2019.8's builtin pe_patch module. Once available patches are known via the above facts, the module will install the patches during the configured patch window.

• For Linux operating systems, this happens through the native Package resource.
• For Windows operating systems, this happens through the patching_as_code::kb class, which comes with this module.
• By default, a reboot is only performed when necessary at the end of a patch run that actually installed patches. You can change this behavior though, to either always reboot or never reboot.
• You can define pre-patch, post-patch and pre-reboot commands for patching runs. We recommend that for Windows, you use Powershell-based commands for these. Specifically for pre-reboot commands on Windows, you must use Powershell-based commands.
• This module will report the configuration for each node in a patching_as_code_config fact

Setup Requirements

To start with patching_as_code, complete the following prerequirements:

• Ensure this module and its dependencies are added to your control repo's Puppetfile
• If you are not running Puppet Enterprise 2019.8.0 or higher, you'll also need to add the albatrossflavour/os_patching module to your control repo's Puppetfile
• If you are running Puppet Enterprise 2019.8.0 or higher, the built-in pe_patch module will be used by default. You can however force the use of the os_patching module if so desired, by setting the optional patching_as_code::use_pe_patch parameter to false. To prevent duplicate declarations of the pe_patch class in PE 2019.8.0+, this module will default to NOT declaring the pe_patch class. This allows you to use the builtin "PE Patch Management" classification groups to classify pe_patch. If you however would like this module to control the classification of pe_patch for you (and sync the patch_group parameter), please set the patching_as_code::classify_pe_patch parameter to true.
• For Linux operating systems, ensure your package managers are pointing to repositories that are publishing new package versions as needed
• For Windows operating systems, ensure Windows Update is configured to check with a valid update server (either WSUS, Windows Update or Microsoft Update). If you want, you can use the puppetlabs/wsus_client module to manage the Windows Update configuration.

Beginning with patching_as_code

To get started with the patching_as_code module, include it in your manifest:

include patching_as_code

or

class {'patching_as_code':}

This enables automatic detection of available patches, and puts all the nodes in the primary patch group. By default this will patch your systems on the 3rd Friday of the month, between 22:00 and midnight (00:00), and perform a reboot if necessary. On PE 2019.8 or newer this will not automatically classify the pe_patch class, so that you can control this through PE's builtin "PE Patch Management" node groups.

To allow patching_as_code to control & declare the pe_patch class, change the declaration to:

class {'patching_as_code':
  classify_pe_patch => true
}

This will change the behavior to also declare the pe_patch class, and match its patch_group parameter with this module's patch_group parameter. In this scenario, make sure you do not classify your nodes with pe_patch via the "PE Patch Management" node groups or other means.

Usage

To control which patch group(s) a node belongs to, you need to set the patch_group parameter of the class. It is highly recommended to use Hiera to set the correct value for each node, for example:

patching_as_code::patch_group: early

The module provides 6 patch groups out of the box:

weekly:    patches each Thursday of the month, between 09:00 and 11:00, performs a reboot if needed
testing:   patches every 2nd Thursday of the month, between 07:00 and 09:00, performs a reboot if needed
early:     patches every 3rd Monday   of the month, between 20:00 and 22:00, performs a reboot if needed
primary:   patches every 3rd Friday   of the month, between 22:00 and 00:00, performs a reboot if needed
secondary: patches every 3rd Saturday of the month, between 22:00 and 00:00, performs a reboot if needed
late:      patches every 4th Saturday of the month, between 22:00 and 00:00, performs a reboot if needed

There are also 2 special built-in patch groups:

always:    patches immediately when a patch is available, can patch in any agent run, performs a reboot if needed
never:     never performs any patching and does not reboot

If you want to assign a node to multiple patch groups, specify an array of values in Hiera:

patching_as_code::patch_group:
  - testing
  - early

or, in flow style:

patching_as_code::patch_group: [ testing, early ]

Note: if you assign a node to multiple patch groups, the value for the patch group provided to pe_patch/os_patching will be a space-separated list of the assigned patch groups. This is because pe_patch/os_patching do not natively support multiple patch groups, so we work around this by converting our list a single string that pe_patch/os_patching can work with. This is purely for cosmetic purposes and does not affect the functionality of either solution.

Customizing the patch groups

You can customize the patch groups to whatever you need. To do so, simply copy the patching_as_code::patch_schedule hash from the data/common.yaml in this module, and paste it into your own Hiera store (recommended to place it in your Hiera's own common.yaml). This Hiera value will now override the defaults that the module provides. Customize the hash to your needs.

The hash has the following structure:

patching_as_code::patch_schedule:
  <name of patch group>:
    day_of_week:   <day to patch systems>
    count_of_week: <the Nth time day_of_week occurs in the month>
    hours:         <start of patch window> - <end of patch window>
    max_runs:      <max number of times that Puppet can perform patching within the patch window>
    reboot:        always | never | ifneeded

For example, say you want to have the following 2 patch groups:

group1: patches every 2nd Sunday of the month, between 10:00 and 11:00, max 1 time, reboots if needed
group2: patches every 3nd and 4th Monday of the month, between 20:00 and 22:00, max 3 times, does not reboot
group3: patches every day in the 3rd week of the month, between 18:00 and 20:00, max 4 times, always reboots

then define the hash as follows:

patching_as_code::patch_schedule:
  group1:
    day_of_week: Sunday
    count_of_week: 2
    hours: 10:00 - 11:00
    max_runs: 1
    reboot: ifneeded
  group2:
    day_of_week: Monday
    count_of_week: [3,4]
    hours: 20:00 - 22:00
    max_runs: 3
    reboot: never
  group3:
    day_of_week: Any
    count_of_week: 3
    hours: 18:00 - 20:00
    max_runs: 4
    reboot: always

Controlling which patches get installed

If you need to limit which patches can get installed, use the blocklist/allowlist capabilties. This is best done through Hiera by defining an array values for patching_as_code::blocklist and/or patching_as_code::allowlist.

To prevent KB2881685 from getting installed:

patching_as_code::blocklist:
  - KB2881685

To only allow the installation of a specific set of 3 KB articles:

patching_as_code::allowlist:
  - KB123456
  - KB234567
  - KB345678

Both options can be combined, in that case the list of available updates first gets reduced to the what is allowed by the allowlist, and then gets further reduced by any blocklisted updates.

Defining situations when patching needs to be skipped

There could be situations where you don't want patching to occur if certain conditions are met. This module supports two such situations:

• A specific process is running that must not be interrupted by patching
• The node to be patched is currently connected via a metered link (Windows only)

Managing unsafe processes for patching

You can define a list of unsafe processes which, if any are found to be active on the node, should cause patching to be skipped. This is best done through Hiera, by defining an array value for patching_as_code::unsafe_process_list.

To skip patching if application1 or application2 is among the active processes:

patching_as_code::unsafe_process_list:
  - application1
  - application2

This works on both Linux and Windows, and the matching is done case-insensitive. If one process from the unsafe_process_list is found as an active process, patching will be skipped.

Managing patching over metered links (Windows only)

By default, this module will not perform patching over metered links (e.g. 3G/4G connections). You can control this behavior through the patch_on_metered_links parameter. To force patching to occur even over metered links, either define this value in Hiera:

patching_as_code::patch_on_metered_links: true

or set this parameter as part of calling the class:

class {'patching_as_code':
  patch_on_metered_links => true
}

Defining pre/post-patching and pre-reboot commands

You can control additional commands that get executed at specific times, to facilitate the patch run. For example, you may want to shutdown specific applications before patching, or drain a kubernetes node before rebooting. The order of operations is as follows: 1) If reboots are enabled, check for pending reboots and reboot system immediately if a pending reboot is found 2) Run pre-patching commands 3) Install patches 4) Run post-patching commands 5) If reboots are enabled, run pre-reboot commands (if a reboot is pending, or when reboots are set to always) 6) If reboots are enabled, reboot system (if a reboot is pending, or when reboots are set to always)

To define the pre/post-patching and pre-reboot commands, you need to create hashes in Hiera. The commands will be executed as Exec resources, and you can use any of the allowed attributes for that resource (just don't use metaparameters). There are 3 hashes you can define:

patching_as_code::pre_patch_commands
patching_as_code::post_patch_commands
patching_as_code::pre_reboot_commands

It's best to define this in Hiera, so that the commands can be tailored to individual nodes or groups of nodes. A hash for a command (let's use pre-reboot as an example) looks like this in Hiera:

patching_as_code::pre_reboot_commands:
  prep k8s for reboot:
    command: /usr/bin/kubectl drain k8s-3.company.local --ignore-daemonsets --delete-local-data

Here's another example, this time for a pre-patch powershell command on Windows:

patching_as_code::pre_patch_commands:
  shutdown SQL server:
    command: Stop-Service MSSQLSERVER -Force 
    provider: powershell

As you can see, it's just like defining Exec resources.

Note that specifically for patching_as_code::pre_reboot_commands, the provider:, onlyif: and unless: parameters will be ignored, as these are overwritten by the internal logic to detect pending reboots. On Linux the provider: is forced to posix, on Windows it is forced to powershell.

Limitations

This solution will patching to initiate whenever an agent run occurs inside the patch window. On Windows, patch runs for Cumulative Updates can take a long time, so you may want to tune the hours of your patch windows to account for a patch run getting started near the end of the window and still taking a significant amount of time.

Reference

Table of Contents

Classes

• patching_as_code: Framework for patch management as code. Works alongside the puppetlabs/pe_patch or albatrossflavour/os_patching modules

Classes

patching_as_code

Framework for patch management as code. Works alongside the puppetlabs/pe_patch or albatrossflavour/os_patching modules

Examples

Using the module with defaults, or controlling options through Hiera

include patching_as_code

Forcing the classification of pe_patch on PE 2019.8.0+

class {'patching_as_code':
  classify_pe_patch => true
}

Forcing the use of albatrossflavour/os_patching on PE 2019.8.0+

class {'patching_as_code':
  use_pe_patch => false
}

Parameters

The following parameters are available in the patching_as_code class.

patch_group

Data type: Variant[String,Array[String]]

Name(s) of the patch_group(s) for this node. Must match one or more of the patch groups in $patch_schedule To assign multiple patch groups, provide this parameter as an array

patch_schedule

Data type: Hash

Hash of available patch_schedules. Default schedules are in /data/common.yaml of this module

Options:

• :day_of_week String: Day of the week to patch, valid options: 'Any', 'Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday'
• :count_of_week Variant[Integer,Array[Integer]]: Which week(s) in the month to patch, use number(s) between 1 and 5
• :hours String: Which hours on patch day to patch, define a range as 'HH:MM - HH:MM'
• :max_runs String: How many Puppet runs during the patch window can Puppet install patches. Must be at least 1.
• :reboot String: Reboot behavior, valid options: 'always', 'never', 'ifneeded'

blocklist

Data type: Array

List of updates to block from installing

allowlist

Data type: Array

List of updates that are allowed to be installed. Any updates not on this list get blocked.

unsafe_process_list

Data type: Array

List of processes that will cause patching to be skipped if any of the processes in the list are active on the system.

pre_patch_commands

Data type: Hash

Hash of command to run before patching

Options:

• :command String: The pre-patching command to execute
• :path String: The path for the command
• :provider String: The provider for the command

post_patch_commands

Data type: Hash

Hash of command to run after patching

Options:

• :command String: The post-patching command to execute
• :path String: The path for the command
• :provider String: The provider for the command

pre_reboot_commands

Data type: Hash

Hash of command to run before rebooting

Options:

• :command String: The pre-reboot command to execute
• :path String: The path for the command

Note: the provider for the command gets forced to posix on Linux and powershell on Windows

plan_patch_fact

Data type: Optional[String]

Reserved parameter for running patching_as_code via a Plan (future functionality).

Default value: undef

enable_patching

Data type: Optional[Boolean]

Controls if patching_as_code is allowed to install any updates. Can be used to disable patching with a single override.

Default value: true

security_only

Data type: Optional[Boolean]

Install only security updates. This works for Linux today, but requires a not-yet shipped update to pe_patch for Windows (this setting has no effect on Windows today)

Default value: false

use_pe_patch

Data type: Optional[Boolean]

Use the pe_patch module if available (PE 2019.8+). Defaults to true.

Default value: true

classify_pe_patch

Data type: Optional[Boolean]

Controls if the pe_patch class (PE 2019.8+) is controlled by this module. When enabled, this module will classify the node with pe_patch, and set it's patch_group according to this module's patch_group. When disabled (default), you can use PE's own "PE Patch Management" groups to classify nodes with pe_patch. In that case, please make sure you match the patch_group variable in pe_patch with the patch_group in patching_as_code

Default value: false

patch_on_metered_links

Data type: Optional[Boolean]

Controls if patches are installed when the active network connection is a metered link. This setting only has affect for Windows operating systems. When enabled, patching are installed even over a metered link. When disabled (default), patches are not installed over a metered link.

Default value: false

Changelog

All notable changes to this project will be documented in this file.

Release 0.6.2

Features

• Adds support for setting the value Any to the day_of_week parameter in a patch schedule

Release 0.6.1

Bugfixes

• Removed the weekday attribute of the schedule resource that this module uses internally to restrict when patches can be applied. In certain edge cases where the Puppet server is in a very different timezone from a managed node, there can be a 1 day date difference between the two systems. This creates a scenario where the node never receives a valid patch schedule. By removing the weekday parameter from the schedule resource, this can no longer occur. Other logic still protects the actual day on which the patching is allowed so this parameter wasn't necessary.

Release 0.6.0

Features

• Adds support for providing an array of values to the patch_group attribute of the patching_as_code class
• Adds support for providing an array of values to the count_of_week parameter in a patch schedule

Release 0.5.0

Features

• Removes dependency on the windows_updates module, we can now install Windows Updates natively
• Adds a Task to install a Windows Update over WinRM or PCP
• Updates the PDK to 2.0.0
• No longer fails the resource if the Windows Update is no longer available/applicable for the node
• Write a patching_as_code_config fact that reports configuration state
• Support security-only patching via a new security_only parameter to the class. This works for Linux today, but requires a not-yet shipped update to pe_patch for Windows
• Preparations for being able to run patching_as_code as a plan, not yet active.

Release 0.4.3

Bugfixes

• Ensure yum-utils package on all RedHat/CentOS versions, not just 8
• Use ensure_packages() for safer enforcement of yum-utils package

Release 0.4.2

Bugfixes

• Account for $facts['operatingsystemmajrelease'] returning a string instead of an integer

Release 0.4.1

Bugfixes

• For parsing the result of /usr/bin/needs-restarting -r in CentOS 7/8, the script was if [ $? -eq 0 ] instead of if [ $? -eq 1 ], which caused the logic to be flipped.

Release 0.4.0

Features

• Completely rewrote the reboot behavior, so that pending reboot detections fully works both before patching and after patching, in the same Puppet run. There is no more dependency on the reboots.reboot_required portion of the pe_patch/os_patching fact, all logic is now internal and no longer requires multiple Puppet runs.
• Changed the default schedules to reboot: ifneeded (was reboot: always), now that the pending reboot logic has improved so much
• Ensured that pre_reboot commands will now trigger when necessary (only one scenario can happen at a time):
  • when an OS pending reboot is detected at the start of a run (before patching)
  • when an OS pending reboot is detected at the end of a run (after patching)
• Forced pre_reboot commands (which are essentially Exec resources) to use the posix provider on Linux and the powershell provider on Windows, so that the pending reboot detection logic can be injected to the resource dynamically.

Release 0.3.0

Features

• Rewrote updating of Linux packages to use a custom type (patch_package), which dynamically updates and/or creates package resources for patching in the catalog on the agent side. This ensures no duplicate package declarations can occur on the server side, due to the parsing-order dependency of defined() and defined_with_params(). Neither of these functions are used anymore.

Release 0.2.9

Bugfixes

• Also protect against duplicate package declarations when ensure is set to a version. This isn't 100% bulletproof as the check is parse-order-dependent, but will work in most cases.

Release 0.2.8

Bugfixes

• Ensured Linux patches cannot cause duplicate declarations

Release 0.2.7

Bugfixes

• Added dependency to puppetlabs/puppet_agent to the module's metadata

Release 0.2.6

Features

• Added a patch_unsafe_process_active custom fact that reflects if any process from the unsafe_process_list parameter was found active on the system.
• Added a unsafe_process_list parameter to the patching_as_code class, which defines processes for the system that must cause patching to be skipped if any of those processes are active. Defaults to an empty array.

Release 0.2.5

Features

• Added cache cleanup for other providers (dnf, apt, zypper)
• Added refresh of pe_patch::fact::exec / os_patching::fact::exec resources, to auto-update patch state after patching. This prevents unneccesary patching runs that perform no updates

Bugfixes

• Increased reboot delay after patching from 1 to 5 minutes, to account for remaining activities in the Puppet run

Release 0.2.4

Bugfixes

• Fixes the datatype of the metered_link fact, this was expected to be Boolean but got reported as a String, causing the logic to break.

Release 0.2.3

Features

• Added a metered_link custom fact that detects metered network connections on Windows
• Added a patch_on_metered_links parameter to the patching_as_code class, which controls if patches are installed when running over a metered link (Windows only). Defaults to false.

Release 0.2.2

Features

• This update ensures that patching_as_code defaults to NOT classify the pe_patch class on PE 2019.8.0, so that you can use the builtin "PE Patch Management" node group(s) to classify pe_patch. Since UI will be further improved in PE for this, it makes sense that this would be the leading way to classify pe_patch. This module can still be given control over pe_patch, as described in the updated Readme.
• The blacklist and whitelist have been renamed to blocklist and allowlist.
• Documentation has been updated, with a reference for the main manifest.

Release 0.2.1

Bugfixes Ensure pre/post-patching & pre-reboot commands use the same schedule

Release 0.2.0

Fixes pending reboot logic, adds pre/post-patching & pre-reboot command support

Features

• Ensures pending reboots are handled correctly, skipping patch installs completely
• Allows defining Exec resources dynamically for pre/post-patching & pre-reboot commands
• Refactors reboot logic into main manifest

Known Issues Tested on Windows 2016 and 2019, and CentOS 7

Release 0.1.0

Initial release

Features

• Integrates with albatrossflavour/os_patching and puppetlabs/pe_patch
• Customizable patch windows
• Patch window based on Nth weekday in the month
• Reboot control
• Yum clean support

Known Issues Tested on Windows 2016 and 2019, and CentOS 7