Puppet Archive

Table of Contents

1. Overview
2. Module Description
3. Setup
4. Usage
   • Example
   • File permission
   • Network files
   • Extract customization
   • S3 Bucket
5. Reference
6. Development

Warning:

Release 0.5.x contains significant changes:

• faraday, faraday_middleware no longer required.
• ruby provider is the default for windows (using net::http).
• archive gem_provider attribute deprecated.
• archive::artifactory server, port, url_path attributes deprecated.
• S3 bucket support (experimental).

Release 0.3.x contains breaking changes

• The parameter 7zip have been changed to seven_zip to conform to Puppet 4.x variable name requirements.
• The namevar name have been changed to path to allow files with the same filename to exists in different filepath.
• This project have been migrated to voxpupuli, please adjust your repo git source.

Experimental:

The following define type/features are experimental and subject to change:

• archive::artifactory
• archive::go
• archive::nexus
• checksum_url (to support nexus files)

Overview

This module manages download, deployment, and cleanup of archive files.

Module Description

This module uses types and providers to download and manage compress files, with optional lifecycle functionality such as checksum, extraction, and cleanup. The benefits over existing modules such as puppet-staging:

• Implemented via types and provider instead of exec resource.
• Follows 302 redirect and propagate download failure.
• Optional checksum verification of archive files.
• Automatic dependency to parent directory.
• Support Windows file extraction via 7zip.
• Able to cleanup archive files after extraction.

Setup

The module requires 7zip for windows clients which is installed via include '::archive'. On posix systems, curl is the default provider. The default provider can be overwritten by configuring resource defaults in site.pp:

Archive {
  provider => 'ruby',
}

Users of the module is responsbile for archive package dependencies for alternative providers and all extraction utilities such as tar, gunzip, bunzip:

if $::facts['osfamily'] != 'windows' {
  package { 'wget':
    ensure => present,
  }

  package { 'bunzip':
    ensure => present,
  }

  Archive {
    provider => 'wget',
    require  => Package['wget', 'bunzip'],
  }
}

Usage

Archive module dependency is managed by the archive class. This is only required for windows platform. By default 7zip is installed via chocolatey, but can be adjusted to use the msi package instead:

class { 'archive':
  seven_zip_name     => '7-Zip 9.20 (x64 edition)',
  seven_zip_source   => 'C:/Windows/Temp/7z920-x64.msi',
  seven_zip_provider => 'windows',
}

Usage Example

include '::archive' # NOTE: optional for posix platforms

archive { '/tmp/jta-1.1.jar':
  ensure        => present,
  extract       => true,
  extract_path  => '/tmp',
  source        => 'http://central.maven.org/maven2/javax/transaction/jta/1.1/jta-1.1.jar',
  checksum      => '2ca09f0b36ca7d71b762e14ea2ff09d5eac57558',
  checksum_type => 'sha1',
  creates       => '/tmp/javax',
  cleanup       => true,
}

archive { '/tmp/test100k.db':
  source   => 'ftp://ftp.otenet.gr/test100k.db',
  username => 'speedtest',
  password => 'speedtest',
}

File permission

When extracting files as non-root user, either ensure the target directory exists with the appropriate permission (see tomcat.pp for full working example):

$dirname = 'apache-tomcat-9.0.0.M3'
$filename = "${dirname}.zip"
$install_path = "/opt/${dirname}"

file { $install_path:
  ensure => directory,
  owner  => 'tomcat',
  group  => 'tomcat',
  mode   => '0755',
}

archive { $filename:
  path          => "/tmp/${filename}",
  source        => 'http://www-eu.apache.org/dist/tomcat/tomcat-9/v9.0.0.M3/bin/apache-tomcat-9.0.0.M3.zip',
  checksum      => 'f2aaf16f5e421b97513c502c03c117fab6569076',
  checksum_type => 'sha1',
  extract       => true,
  extract_path  => '/opt',
  creates       => "${install_path}/bin",
  cleanup       => true,
  user          => 'tomcat',
  group         => 'tomcat',
  require       => File[$install_path],
}

or use an subscribing exec to chmod the directory afterwards:

$dirname = 'apache-tomcat-9.0.0.M3'
$filename = "${dirname}.zip"
$install_path = "/opt/${dirname}"

file { '/opt/tomcat':
  ensure => 'link',
  target => $install_path
}

archive { $filename:
  path          => "/tmp/${filename}",
  source        => "http://www-eu.apache.org/dist/tomcat/tomcat-9/v9.0.0.M3/bin/apache-tomcat-9.0.0.M3.zip",
  checksum      => 'f2aaf16f5e421b97513c502c03c117fab6569076',
  checksum_type => 'sha1',
  extract       => true,
  extract_path  => '/opt',
  creates       => $install_path,
  cleanup       => 'true',
  require       => File[$install_path],
}

exec { 'tomcat permission':
  command   => "chown tomcat:tomcat $install_path",
  path      => $::path,
  subscribe => Archive[$filename],
}

Network files

For large binary files that needs to be extracted locally, instead of copying the file from the network fileshare, simply set the file path to be the same as the source and archive will use the network file location:

archive { '/nfs/repo/software.zip':
  source        => '/nfs/repo/software.zip'
  extract       => true,
  extract_path  => '/opt',
  checksum_type => 'none', # typically unecessary
  cleanup       => false,  # keep the file on the server
}

Extract Customization

The extract_flag or extract_command parameters can be used to override the default extraction command/flag (defaults are specified in achive.rb).

# tar striping directories:
archive { '/var/lib/kafka/kafka_2.10-0.8.2.1.tgz':
  ensure          => present,
  extract         => true,
  extract_command => 'tar xfz %s --strip-components=1',
  extract_path    => '/opt/kafka_2.10-0.8.2.1',
  cleanup         => true,
  creates         => '/opt/kafka_2.10-0.8.2.1/config',
}

# zip freshen existing files (zip -of %s instead of zip -o %s):
archive { '/var/lib/example.zip':
  extract      => true,
  extract_path => '/opt',
  extract_flag => '-of',
  cleanup      => true,
  subscribe    => ...,
}

S3 bucket

S3 support is implemented via the AWS CLI. By default this dependency is only installed for Linux VMs running on AWS, or enabled via aws_cli_install option:

class { '::archive':
  aws_cli_install => true,
}

# See AWS cli guide for credential and configuration settings:
# http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
file { '/root/.aws/credential':
  ensure => file,
  ...
}

archive { '/tmp/gravatar.png':
  ensure => present,
  source => 's3://bodecoio/gravatar.png',
}

NOTE: Alternative s3 provider support can be implemented by overriding the s3_download method:

Reference

Classes

• archive: install 7zip package (Windows only) and aws cli for s3 support.
• archive::staging: install package dependencies and creates staging directory for backwards compatibility. Use the archive class instead if you do not need the staging directory.

Define Resources

• archive::artifactory: archive wrapper for JFrog Artifactory files with checksum.
• archive::go: archive wrapper for GO Continuous Delivery files with checksum.
• archive::nexus: archive wrapper for Sonatype Nexus files with checksum.

Resources

Archive

• ensure: whether archive file should be present/absent (default: present)
• path: namevar, archive file fully qualified file path.
• filename: archive file name (derived from path).
• source: archive file source, supports http|https|ftp|file|s3 uri.
• username: username to download source file.
• password: password to download source file.
• cookie: archive file download cookie.
• checksum_type: archive file checksum type (none|md5|sha1|sha2|sh256|sha384|sha512). (default: none)
• checksum: archive file checksum (match checksum_type)
• checksum_url: archive file checksum source (instead of specify checksum)
• checksum_verify: whether checksum will be verified (true|false). (default: true)
• extract: whether archive will be extracted after download (true|false). (default: false)
• extract_path: target folder path to extract archive.
• extract_command: custom extraction command ('tar xvf example.tar.gz'), also support sprintf format ('tar xvf %s') which will be processed with the filename: sprintf('tar xvf %s', filename)
• extract_flags: custom extraction options, this replaces the default flags. A string such as 'xvf' for a tar file would replace the default xf flag. A hash is useful when custom flags are needed for different platforms. {'tar' => 'xzf', '7z' => 'x -aot'}.
• user: extract command user (using this option will configure the archive file permission to 0644 so the user can read the file).
• group: extract command group (using this option will configure the archive file permission to 0644 so the user can read the file).
• cleanup: whether archive file will be removed after extraction (true|false). (default: true)
• creates: if file/directory exists, will not download/extract archive.
• proxy_server: specify a proxy server, with port number if needed. ie: https://example.com:8080.
• proxy_type: proxy server type (none|http|https|ftp)

Archive::Artifactory

• path: fully qualified filepath for the download the file or use archive_path and only supply filename. (namevar).
• ensure: ensure the file is present/absent.
• url: artifactory download url filepath. NOTE: replaces server, port, url_path parameters.
• server: artifactory server name (deprecated).
• port: artifactory server port (deprecated).
• url_path: artifactory file path http:://{server}`:{port}/artifactory/{url_path} (deprecated).
• owner: file owner (see archive params for defaults).
• group: file group (see archive params for defaults).
• mode: file mode (see archive params for defaults).
• archive_path: the parent directory of local filepath.
• extract: whether to extract the files (true/false).
• creates: the file created when the archive is extracted (true/false).
• cleanup: remove archive file after file extraction (true/false).

Archive::Artifactory Example:

$dirname = 'gradle-1.0-milestone-4-20110723151213+0300'
$filename = "${dirname}-bin.zip"

archive::artifactory { $filename:
  archive_path => '/tmp',
  url          => "http://repo.jfrog.org/artifactory/distributions/org/gradle/${filename}",
  extract      => true,
  extract_path => '/opt',
  creates      => "/opt/${dirname}",
  cleanup      => true,
}

file { '/opt/gradle':
  ensure => link,
  target => "/opt/${dirname}",
}

Archive::Nexus

Archive::Nexus Example:

archive::nexus { '/tmp/jtstand-ui-0.98.jar':
  url        => 'https://oss.sonatype.org',
  gav        => 'org.codehaus.jtstand:jtstand-ui:0.98',
  repository => 'codehaus-releases',
  packaging  => 'jar',
  extract    => false,
}

Development

We highly welcome new contributions to this module, especially those that include documentation, and rspec tests ;) but will happily guide you through the process, so, yes, please submit that pull request!

Note: If you are writing a dependent module that include specs in it, you will need to set the puppetversion fact in your puppet-rspec tests. You can do that by adding it to the default facts of your spec/spec_helper.rb:

RSpec.configure do |c|
  c.default_facts = { :puppetversion => Puppet.version }
end

##2016/03/18 - Releasing 0.5.1

• GH-146 Set aws_cli_install default to false
• GH-142 Fix wget cookie options
• GH-114 Document extract customization options
• open file in binary mode when writing files for windows download

##2016/03/10 - Releasing 0.5.0

• GH-55 use net::http to stream files
• Add additional documentation
• Simplify duplicate code in download/content methods
• Pin rake to avoid rubocop/rake 11 incompatibility
• Surface "checksum_verify" parameter in archive::nexus
• GH-48 S3 bucket support

##2016/3/2 - Releasing 0.4.8

• VoxPupuli Release
• modulesync to fix forge release issues.
• cosmetic changes due to rubocop update.

##2016/3/1 - Releasing 0.4.7

• VoxPupuli Release
• raise exception when error occurs during extraction.

##2016/2/26 - Releasing 0.4.6

• VoxPupuli Release

##2016/2/26 - Releasing 0.4.5

• Puppet-community release
• Update travis/forge badge location
• Fix aio-agent detection
• Support .gz .xz format
• Fix local files for non faraday providers
• Fix GH-77 allows local files to be specified without using file:///
• Fix GH-78 allow local file:///c:/... on windows
• Fix phantom v0.4.4 release.

##2015/12/2 - Releasing 0.4.4

• Puppet-community release
• Ignore files properly for functional release
• Add authentication to archive::nexus
• Create directory before transfering file
• Refactor file download code
• Create and use fact for archive_windir
• Cleanup old testing code

##2015/11/25 - Releasing 0.4.3

• Puppet-community release

##2015/11/25 - Releasing 0.4.1

• Automate release :)

##2015/11/25 - Releasing 0.4.0

• Migrate Module to Puppet-Community
• Make everything Rubocop Clean
• Make everything lint clean
• Various fixes concerning Jar handling
• Support for wget
• Spec Tests for curl
• Support for bzip
• More robust handling of sha512 checksums

##2015/4/23 - 0.3.0

• Fix Puppet 4 compatability issues
• Fix archive namevar to use path

##2015/3/5 - 0.2.2

• add FTP and File support

##2015/2/26 - 0.2.1

• fix ruby 1.8.7 syntax error

##2015/2/23 - 0.2.0

• fix custom flags options
• add msi installation option for 7zip
• add support for configuring extract command user/group
• use temporary filepath for download

##2014/12/08 - 0.1.8

• Update documentation
• puppet-lint, metadata.json cleanup

##2014/11/13 - 0.1.7

• Fix Puppet Enterprise detection
• Fix checksum length restriction
• Add puppetlabs stdlib/pe_gem dependency
• Add spec testing

##2014/11/05 - 0.1.6

• Fix Windows SSL authentication issues

##2014/11/04 - 0.1.5

• Add cookie support

##2014/10/03 - 0.1.4

• Fix file overwrite and re-extract

##2014/10/03 - 0.1.3

• Fix windows x86 path bug

##2014/10/02 - 0.1.2

• Fix autorequire and installation of dependencies

##2014/10/01 - 0.1.1

• Add windows extraction support via 7zip

##2014/9/26 - 0.1.0

• Initial Release