cfbackup

System backup & restore management
CodingFuture

CodingFuture

codingfuture

3,449 downloads

1,311 latest version

5.0 quality score

Version information

  • 1.3.2 (latest)
  • 1.3.1
  • 1.3.0
released Jan 6th 2020
This version is compatible with:
  • Puppet Enterprise 2019.8.x, 2019.7.x, 2019.5.x, 2019.4.x, 2019.3.x, 2019.2.x, 2019.1.x, 2019.0.x, 2018.1.x, 2017.3.x, 2017.2.x, 2017.1.x, 2016.5.x, 2016.4.x
  • Puppet >=4.7.0 <7.0.0
  • Debian
    ,
    Ubuntu

Start using this module

Tags: backup, sftp, s3, gpg

Documentation

codingfuture/cfbackup — version 1.3.2 Jan 6th 2020

cfbackup - System backup & restore management

Description

Generic configuration of system backup:

  • Remote locations:
    • local - for NFS-like mounts (noop)..
    • sftp - for SSH-based uploads/downloads.
    • s3 - for S3 uploads/downloads.
  • Encryption:
    • asymmetric multi-recipient GPG.
    • ensures high data privacy.
    • allows recovery via offsite secure keys.
    • allows recovery via local host key.
    • allows sharing remote backup location with other hosts without jeopardizing data privacy.
  • Periodic backup strategy:
    • service-specific backup and restore procedures are implemented in other modules.
    • timer-based periodic whole system backup is run (hourly by default).
    • timer-based periodic upload is run (daily by default).
    • periodic backup is aware of base and incremental backup approaches.
    • automatic cleanup of local backups.
    • proper file locking is ensured.
  • File backup strategy:
    • use rdiff-backup for base & incremental periodic dumps.

Terminology & Concept

There are three major types of data for backup:

  • Databases:
    • requires custom backup procedures for efficiency and integrity purposes.
  • Filesystem:
    • raw file copying is acceptable.
  • Applications:
    • dummy backup may help with fast recovery, but proper restoration procedure is just a new deployment.

Locations:

  • /mnt/backup - root for all local backups:
    • desired to be a separate partition or an efficient network mount.
  • /mnt/backup/{namespace}/{id}/ - root of service backups:
    • {namespace} - service namespace, e.g. cfdb.
    • {id} - service ID in the namespace, e.g. CFDB cluster name.
  • /mnt/backup/{namespace}/{id}/{base-datetime} - location of base backups.
    • incremental backup data is assumed to be inside.
  • /mnt/backup/{namespace}/{id}/last_backup - hold value of the last {base-datetime}.
  • {remote}/{hostname}/{namespace}/{id}/ - root for remote backups.

Primary CLI commands:

  • cfbackup_all - perform local backup of services configured via cfbackup::path
  • cfbackup_upload_all - perform upload and cleanup of local backups.
  • cfbackup_download_latest [path] - download the latest available backup for later local restore.

System services:

  • cftimer-backupall - run cfbackup_all`
  • cftimer-uploadall - run cfbackup_upload_all`

Helpers CLI commands:

  • cfbackup_encrypt_helper and cfbackup_decrypt_helper - encryption processing.
  • cfbackup_upload_helper and cfbackup_download_helper - remote transfer processing.
  • cfbackup_periodic_helper.sh - common logic to perform a single service backup.
  • cfrestore_periodic_helper.sh - common logic to perform a single service restore.

Technical Support

Setup

Up to date installation instructions are available in Puppet Forge: https://forge.puppet.com/codingfuture/cfbackup

Please use librarian-puppet or cfpuppetserver module to deal with dependencies.

There is a known r10k issue RK-3 which prevents automatic dependencies of dependencies installation.

Examples

Please check codingufuture/puppet-test for example of a complete infrastructure configuration and Vagrant provisioning.

classes:
    - cfbackup

cfbackup::gpg::recipients:
    - |
        Offsite public key 1
    - |
        Offsite public key 2

# SFTP remote upload
#-------------------
cfbackup::type: sftp
cfbackup::sftp::user: sftp_user
cfbackup::sftp::host: sftp.example.com
cfbackup::sftp::port: 22
cfbackup::sftp::ssh_key: |
    -----BEGIN OPENSSH PRIVATE KEY-----
    ...
    -----END OPENSSH PRIVATE KEY-----

# S3 remote upload
#-------------------
cfbackup::type: s3
cfbackup::s3::url: https://s3.example.com
cfbackup::s3::bucket: backup.example.com
cfbackup::s3::access_key: test123
cfbackup::s3::secret_key: test234

Implicitly created resources

# for SFTP
#------------------
cfnetwork::describe_service:
    cfbackup_sftp:
        server: "tcp/${port}"
cfnetwork::client_port:
    "any:cfbackup_sftp":
        user: root

cfbackup parameters

class cfbackup

  • Enum['s3', 'sftp', 'local'] $type = 'local' - type of remote transfer.
  • Enum['gpg'] $crypt = 'gpg' - type of encryption.
  • String[1] $root_dir = '/mnt/backup' - local location for periodic backups.
  • String[1] $backup_timer = 'hourly' - systemd OnCalendar configuration for cftimer-backupall.
  • String[1] $upload_timer = 'daily' - systemd OnCalendar configuration for cftimer-uploadall.
  • Integer[0] $reserve_ram = 64 - amount of system memory to reserve for backup processing.

resource cfbackup::path

This resource type should be used to register service backup procedures.

  • String[1] $namespace - service namespace.
  • String[1] $id - unique service identifier in the namespace.
  • Enum['files', 'periodic_dump'] $type - type of backup.
  • Boolean $compress = true - should compression be applied (bzip2).
  • Optional[String[1]] $prepare = undef - command to execute during cfbackup_all.

class cfbackup::gpg

Backup encryption and decryption configuration.

  • Array[String[1]] $recipients = [] - list of additional public keys for encryption.
  • Optional[String[1]] $own_key = undef - forced private key for local host.
  • String[1] $own_key_id = $::facts['fqdn'] - identifier to use for local host key.

class cfbackup::sftp

Remote SFTP endpoint configuration.

  • String[1] $user - SSH user name
  • String[1] $host - SSH host
  • String[1] $ssh_key - private SSH key
  • Cfnetwork::Port $port = 22 - SSH port

class cfbackup::s3

Remote S3 endpoint configuration.

  • String[1] $url - S3 URL
  • String[1] $bucket - S3 bucket
  • String[1] $access_key - S3 access key
  • String[1] $secret_key - S3 secret key
  • String[1] $s3cmd_version = 'latest' - version of s3cmd to use