cfsystem

Description

Configure a bare minimal production system regardless of its purpose. It depends on more specific cfnetwork, cfauth and cffirehol modules.

What it does:

• Whatever cfnetwork does
• Whatever cfauth does
• Whatever cffirehol does
• Setups APT for Debian and Ubuntu
• Setups timezone
• Setups hostname based on certname
• Adds firewall rules as required
• Setups special location/pool facts for hiera lookup (see cfsystem::hierapool below)
• Setups email system
• Setups NTP
• Setup all locales and the default locale (configurable)
• Installs many handy system tools which almost any admin would expect
• Forces noop scheduler on SSDs and virtual devices (in guests)
• Forces custom I/O scheduler for real spinning HDDs (deadline by default)
• Adds custom rc.local commands, if needed
• Adds cron job to check if running kernel version matches the latest installed (reboot reminder)
• Ruby framework for other cf* modules
• The following helper scripts are installed
  • cf_clear_email_queue - clear all emails in exim queue
  • cf_clear_frozen_emails - clear only frozen emails in exim queue
  • cf_send_test_email - send test email to admin address
  • cf_kernel_version_check - check if kernel version mismatch the latest installed one
  • cf_auto_block_scheduler - setup auto-detected I/O scheduler per block device
• Public API for Puppet parser:
  • cf_query_facts - wrapper around query_facts
  • cf_query_nodes - wrapper around query_nodes
  • cf_query_resources - wrapper around query_resources
  • cf_stable_sort(arg) - deep sort of Hash/Array to avoid isomorphic configuration "change"

Technical Support

• Example configuration
• Commercial support: support@codingfuture.net

Setup

Please use librarian-puppet or cfpuppetserver module to deal with dependencies.

There is a known r10k issue RK-3 which prevents automatic dependencies of dependencies installation.

Examples

Please check codingufuture/puppet-test for example of a complete infrastructure configuration and Vagrant provisioning.

Implicitly created resources

cfnetwork::describe_services:
    puppet:
        server: 'tcp/8140'
    smtp:
        server: 'tcp/25'
    cfsmtp:
        server:
            - 'tcp/25'  # smtp
            - 'tcp/465' # smtps
            - 'tcp/587' # submission
    # if $cfsystem::add_repo_cacher
    'apcng':
        server: 'tcp/3142'
    # if $cfsystem::repo_proxy
    'aptproxy':
        server: "tcp/${proxy_port}"
cfnetwork::service_ports:
    # foreach $cfsystem::email::listen_ifaces
    "${listen_ifaces}:smtp:cfsystem": {}
    'local:smtp:cfsystem': {}
    # if $cfsystem::add_ntp_server
    "${cfsystem::service_face}:ntp": {}
    # if $cfsystem::add_repo_cacher
    "${cfsystem::service_face}:apcng:cfsystem": {}
    # if ${cfsystem::service_face} not in ['any', 'local']
    'local:apcng:cfsystem': {}
cfnetwork::client_ports:
    'any:puppet:cfsystem':
        user: 'root'
    'local:smtp:cfsystem': {}
    # if $smarthost = undef then dst filtering is disabled
    'any:cfsmtp:cfsystem':
        user => ['root', 'Debian-exim'],
        dst  => $smarthost
    'any:ntp:cfsystem':
        user => ['root', 'ntpd'],
    # if $cfsystem::add_repo_cacher
    'any:http:apcng':
        user: 'apt-cacher-ng'
    # if $cfsystem::add_repo_cacher
    'any:https:apcng':
        user: 'apt-cacher-ng'
    # if $cfsystem::repo_proxy
    'any:aptproxy:cfsystem':
        dst: $proxy_host
        user: 'root'
    # if not $cfsystem::repo_proxy
    'any:http:cfsystem':
        user: 'root'
    # if not $cfsystem::repo_proxy
    'any:https:cfsystem':
        user: 'root'

Class parameters

cfsystem class

• allow_nfs = false - purge RPC packages unless true
• admin_email = undef - email address to use for root and as the default sink
• repo_proxy = undef - if set, use the config as HTTP/HTTPS proxy for package retrieval.
  • host - IP or hostname
  • port - TCP port
• add_repo_cacher = false - if true, install apt-cacher-ng and accept clients on $service_face
• service_face = 'any' - interface to accept client for NTP and HTTP proxy, if enabled separately
• ntp_servers = [ 'pool.ntp.org' ] - upstream NTP server
• add_ntp_server = false - if true, accept NTP service clients on $service_face
• timezone = 'Etc/UTC' - setup system timezone
• apt_purge - passed to apt::purge, purge all sources and preferences by default
• apt_update - passed to apt::update, update daily with 300 second timeout by default
• apt_pin = 1001 - default priority (>=1001 - force downgrades to make the system consistent)
• apt_backports_pin = 600 - default priority (>=1001 - force downgrades to make the system consistent)
• real_hdd_scheduler - default scheduler for not SSD and not virtualized HDDs
• rc_local - list of additional commands to add to /etc/rc.local (SSD and virtual is always 'noop')
• puppet_host = "puppet.${::trusted['domain']}" - Puppet Server hostname
• puppet_cahost = $puppet_host - Puppet CA hostname
• puppet_env = $::environment - Puppet environment
• puppet_use_dns_srv = false - enable support DNS SRV records instead of hostnames
• mcollective = false - controls if mcollective service is enabled
• locale = 'en_US.UTF-8' - default system locale
• reserve_ram = 64 - amount of ram to reserve for system in automatic calculations

cfsystem::hierapool class

Automatically including by cfsystem. This values are useful in hiera.yaml configuration to setup hierarchy based on location and tenant/server pool in it. Example:

    ---
    :backends:
    - yaml
    :hierarchy:
    - "%{::trusted.domain}/%{::trusted.hostname}"
    - "%{::trusted.domain}"
    - "%{::cf_location}/%{::cf_location_pool}"
    - "%{::cf_location}"
    - common
    :merge_behavior: deeper
    :yaml:
    :datadir:

• location = undef - if set, saved into /etc/cflocation
• pool = undef - if set, aved into /etc/cflocationpool

cfsystem::email class

Setup email server for outgoing emails. Please not that this configuration is not intended to accept internet traffic.

• smarthost = undef - if set, use as smarthost to relay outgoing emails through
• smarthost_login = undef - if set, use as login on smarthost
• smarthost_password = undef - if set, use as password on smarthost (plain text)
• relay_nets = <private subnets> - allowed clients for SMTP relay, if relay is enabled with $listen_ifaces
• listen_ifaces = undef - list of interface (cfnetwork::iface names), besides lo to listen for SMTP client relay
• disable_ipv6 = true - if true, IPv6 supports gets disabled (most likely you need it disabled for SMTP)

cfsystem::sysctl class

Setup sysctl entries.

• vm_swappiness = 1 - 0-100 (%) minimize swap activity by default

cfsystem::debian class

Debian-specific configuration.

• apt_url = 'http://httpredir.debian.org/debian' - APT base URL for Debian repos
• security_apt_url = 'http://security.debian.org/' - APT base URL for Debian security repo
• release = 'jessie' - Debian release name to configure

cfsystem::ubuntu class

Ubuntu-specific configuration.

• apt_url = 'mirror://mirrors.ubuntu.com/mirrors.txt' - APT base URL for Ubuntu repos
• release = 'wily' - Ubuntu release name to configure

cfsystem::debian::debconf type

• package = $title - package to configure & install
• ensure = present - passed to package ensure
• config = [] - config entries for debconf-set-selections

cfsystem::dotenv type

A special helper to create entries in user ~/.env files

• user - previously defined user{ $user: home => 'path'} ($home must be explicitly set)
• variable - variable name
• value - value
• env_file = '.env' - name of .env file relative to $home

cfsystem::puppetpki type

Make actual Puppet PKI (CA, CRL, client cert and private key) data available to specific user. By default the data is copied under ~/pki/puppet/.

• user = $title - local user to use
• copy_key = true - if true then private keys of local machine are copied as well
• pki_dir = undef - override the default destination folder

cfsystem::haproxy class

Setup haproxy package. No configuration. Used by other modules

• $disable_standard = true - controls if default HAProxy service must be disabled

Change Log

All notable changes to this project will be documented in this file. This project adheres to Semantic Versioning.

0.9.24

• Added internal cfsystem_info helper to store arbitrary info in cfsystem.json

0.9.23

• Fixed to install libssl1.0.0 dep for latest HAProxy @ Jessie

0.9.22

• Fixed case of PuppetLabs PGP key auto-update without proxy

0.9.21

• Fixed wrong version of PuppetLabs PGP key auto-update getting in release

0.9.20

• Implemented auto-update of PuppetLabs PGP key

0.9.19

• Fixed to enable services during creation in Ruby framework

0.9.18

• Minor fix of private Ruby infrastructure

0.9.17

• Changed parser helper cf_genpass and cf_genport to use facts and act like client-side counterpart
• Added cf_genport helper integrated with facts
• Improved logic of CfSystem.genPort()
• Added cfsystem::haproxy to setup packages
• Added custom $pki_dir support to cfsystem::puppetpki
• Fixed to make sort cfsystem.json sections are sorted as well
• Fixed not to fail all resources, if some resource save handler fails in cfsystem.json
• Added wrappers around puppetdbquery module
• Fixed to support static catalog (no puppet:// source)

0.9.16

• Added control parameter for mcollective service
• Improved security of .env files - only owner can read
• Implemented stable sorting of cfsystem.json section content
• Fixed exim4 provisioning deps & misc.

0.9.15

• Fixed to correctly support Ubuntu Xenial
  • Added disabling of IPv6 in APT
  • Added disabling of not yet supported backports
  • Changed to use fixed mirror by default

0.9.14

• Disabled scheduled agent runs for safety purposes
• Implemented framework support for systemd slices

0.9.13

• Fixed to pass strict mode checking
• Implemented automatic memory distribution with incremental part definitions per service
• Added cfsystem::puppetpki type to copy puppet PKI for local user
• Added strace to list of standard tools
• Updated deps to latest versions

0.9.12

• Workaround to use jessie for stretch for PuppetLabs APT repo
• Changed back to use xenial for appeared PuppetLabs APT repo
• Added support for next Ubuntu 16.10 (yakkety)
• Implemented experimental framework for:
  • weight based memory distribution
  • resource configuration management
• Implemented a new feature cfsystem::dotenv to manange ~/.env config
• Moved block scheduler logic from rc.local to cf_auto_block_scheduler script

0.9.11

• Added missing apt-listchanges installation
• Added a workaround to install wily packages for xenial until PuppetLabs release those
• Added special '_apt' user support for stretch/xenial
• Updated to use current Debian/Ubuntu release (fact) as the default for APT

0.9.10

• Fixed cf_kernel_version_check to work on Ubuntu with /proc/version_signature

0.9.9

• Implemented cron job for outdated kernel version detection (reboot reminder)
• Added generic /opt/codingfuture/bin folder for all installed scripts
• Moved to generic bin dir and renamed exim helper tools
  • cf_send_test_email
  • cf_clear_email_queue
  • cf_clear_frozen_emails

0.9.8

• Added generic infrastructure for Debconf support (cfsystem::debian::debconf)
• Added support for default system locale
• Added installation of all locales
• Updated Timezone configuration to properly utilize Debconf on Debian & Ubuntu
• Added APT pinning support with forced downgrades by default
• Fixed apt-cacher-ng to allow root user http/https connections during dpkg processing

0.9.7

• Fixed use_srv_records puppet setting to depend on correct parameter
• Fixed to unconditionally install puppet-agent package

0.9.6

• Fixed issue of ca_server not being properly set in some cases

0.9.5

• Changed to force 'default' value for cf_location and cf_location_pool, unless set. That's required to minimize issues due to empty interpolation in Hiera paths.
• Moved sudo and openssh-server installation to cfauth module
• Reorganized internal manifests
• Added puppet agent configuration parameters, including CA server, use DNS SRV records, and puppet environment
• Dropped off external timezone module dependency and re-implemented internally
• Changed to use PuppetLabs approved augeas sysctl module
• Dropped of external openntpd module dependency and re-implementd internally due to original implementation dependency on module_data module which breaks Puppet 4.
• OpenNTPd is using "servers" instead of "server" configuration option now.

0.9.4

• Removed inittab processing for Xen PV guests as they should use systemd

0.9.3

• Force to re-execute sysctl conf in rc.local
• Added custom I/O scheduler support
• Forced noop scheduler for SSD and virtual devices
• Added custom rc.local commands support
• Added 'cf_virt_detect' which has output of systemd-detect-virt
• Fixed issue of apt-cacher-ng bootstrap when APT config depends on not yet installed proxy
• Fixed to use xen PV console on xen hosts

0.9.2

• Added hiera.yaml version 4 support

0.9.1

• Added APT purge and update control through cfsystem parameters

0.9.0

Initial release