Forge Home

clustersync

Combines lsyncd and csync2 to sync files across a cluster.

10,099 downloads

9,683 latest version

3.6 quality score

We run a couple of automated
scans to help you access a
module's quality. Each module is
given a score based on how well
the author has formatted their
code and documentation and
modules are also checked for
malware using VirusTotal.

Please note, the information below
is for guidance only and neither of
these methods should be considered
an endorsement by Puppet.

Version information

  • 1.0.1 (latest)
  • 1.0.0
released Feb 16th 2015

Start using this module

  • r10k or Code Manager
  • Bolt
  • Manual installation
  • Direct download

Add this module to your Puppetfile:

mod 'scottsb-clustersync', '1.0.1'
Learn more about managing modules with a Puppetfile

Add this module to your Bolt project:

bolt module add scottsb-clustersync
Learn more about using this module with an existing project

Manually install this module globally with Puppet module tool:

puppet module install scottsb-clustersync --version 1.0.1

Direct download is not typically how you would use a Puppet module to manage your infrastructure, but you may want to download the module in order to inspect the code.

Download

Documentation

scottsb/clustersync — version 1.0.1 Feb 16th 2015

scottsb-clustersync

Overview

This module allows you to easily combine lsyncd with csync2 to sync files across a set of servers. Lsyncd watches the file system, triggering csync2 on a change, which then copies the files out to the other machines. Csync2 supports multiple servers (not just a pair), intelligently handling conflicts and file deletions.

This solution works well for any number of files (into the tens of millions), but it does not work well for files that change frequently. For this you need a distributed filesystem like GlusterFS.

The core functionality is provided by Matthias Saou's modules thias-lsyncd and thias-csync2. This module simply serves as a convenience wrapper (which Matthew left as an exercise for the reader). The primary benefits are:

  • Streamlined syntax.
  • No need to repeat sync directories for both lsyncd and csync2.

Usage

There must be one clustersync declaration and one or more clustersync::serverset declarations.

clustersync

The clustersync class defines all directories that need to be synced as well as global options for the csync2 and lsyncd daemons. Parameters:

  • sources: hash where the key is a path to sync and the value is the name of a serverset declaration
  • logdir_owner: lsyncd log file owner
  • logdir_group: lsyncd log file group
  • logdir_mode: lsyncd log file permissions
  • csync_port: port csync2 will listen on
  • csync_only_from: IPs csync2 will accept connections from (CIDR notation)
  • lsyncd_template: override of lsyncd config template

Note: This module assumes that you have a file resource declared for each of the sources keys. The file resource should have the same name as the path.

clustersync::serverset

The clustersync::serverset resource matches up to one or more values from the sources hash in the clustersync declaration. It specifies the servers that the indicated directory will be synced to. Parameters:

  • servers: array of hosts to sync with*
  • key_source: path to the pre-shared key (mutually exclusive with key_content)**
  • key_content: content of the pre-shared key (mutually exclusive with key_source)
  • csync2_template: override of csync2 config template

* Hostnames must match the output of the hostname command. An IP address may optionally be specified in addition using the syntax hostname@ipaddress. ** A pre-shared key can be generated with the command csync2 -k filename.

Example

# Set up source directories.
file { [
  '/var/uploads',
  '/etc/serverd',
]:
  ensure => directory,
}

# Set up the automatic sync of dynamically uploaded files.
class { 'clustersync':
  sources         => {
    '/var/uploads' => 'uploads',
    '/etc/serverd' => 'serverd',
  },
  csync_only_from => '10.0.100.0/24 10.0.200.0/24'
}
clustersync::serverset { 'uploads':
  servers => [
    'foo.example.net@10.0.100.1',
    'bar.example.net@10.0.100.2',
    'baz.example.net@10.0.100.3',
  ],
  key_source => 'puppet:///mnt/csync2_uploads.key'
}
clustersync::serverset { 'serverd':
  servers => [
    'barium.example.net',
    'sodium.example.net',
  ],
  key_source => 'puppet:///mnt/csync2_serverd.key'
}

Limitations

This module has only been tested with RHEL 6.5, though it should work on any Linux distributions that include both csync2 and lsyncd in their package managers. Most OS-specific support will fall to the underlying thias-lsyncd and thias-csync2 modules, but if you experience distro-specific issues, raise them on GitHub.

TODO

These are known areas that need to be improved:

  • Add automated testing beyond just syntax check.
  • Decouple from thias-xinetd so that any xinetd module can be used.
  • Enforcement of option types and mutually exclusive options.