Version information
This version is compatible with:
- RedHat,CentOS,Scientific
Start using this module
Documentation
puppet-cvmfs
This cvmfs module is designed to install, enable and configure CvmFS clients and servers.
For general details on CvmFS see http://cernvm.cern.ch/portal/filesystem
Custom Facts
The module include one customfacts
- cvmfspartsize returns the size in megabytes of partition that contains the CVMFS_CACHE_BASE.
Client Configuration
To configure a cvmfs client to mount cvmfs repository with the default autofs.
class{"cvmfs":
cvmfs_http_proxy => 'http://ca-proxy.example.org:3128',
cvmfs_quota_limit => 100
}
cvmfs::mount{'files.example.org:
cvmfs_server_url => 'http://web.example.org/cvmfs/files.example.org',
}
To configure a cvmfs client to mount a domain of repositories with autofs.
class{"cvmfs":
cvmfs_http_proxy => 'http://ca-proxy.example.org:3128',
cvmfs_quota_limit => 100,
}
cvmfs::domain{'example.net'
cvmfs_server_url => 'http://web.example.org/cvmfs/@fqrn@'
}
To use puppet's mount type rather that autofs a typical configuration might be the following. This examples configures a cvmfs domain, a configuration repository and finally a particular repository for mount.
class{'::cvmfs':
mount_method => 'mount',
}
cvmfs::domain{'example.org':
cvmfs_server_url => 'http://web.example.org/cvmfs/@fqrn@'
}
cvmfs::mount{'cvmfs-config.example.org':
require => Cvmfs::Domain['example.org'],
}
cvmfs::mount{'myrepo.example.org':
require => Cvmfs::Domain['example.org'],
}
Parameters to Cvmfs Class
config_automounter
now deprecated, setting this will cause a msg and fail.mount_method
A string , can be set to autofs, mount or none. The default autofs will configure cvmfs to be mounted with autofs. The mount option will use puppets mount type, currently adding a line to /etc/fstab. The none option skips all mounting. Note that migrating between for instance autofs and then mount is not supported.manage_autofs_service
boolean defaults to true, should the autofs service be maintained.cvmfs_quota_limit
The cvmfs quota size in megabytes. See params.pp for default.cvmfs_quota_ratio
If set to ration, e.g '0.8' then 0.8 of the partition size the cvmfs cache is on will be used. Setting this assumes you have allocated a partition to cvmfs cache.cvmfs_http_proxy
List of squid servers, see params.pp for default.cvmfs_cache_base
Location of the CVMFS cache base, see params.pp for default.cvmfs_dns_min_ttl
Minimum ttl of DNS lookups.cvmfs_dns_max_ttl
Maximum ttl of DNS lookups.cvmfs_claim_ownership
Whether the client claims ownership of files or not, see params.pp for default.cvmfs_memcache_size
Size of the CernVM-FS meta-data memory cache in Megabyte.cvmfs_mount_rw
Mount option to mount read-only or read/write, 'yes|no', see params.pp for default.cvmfs_follow_redirects
Sets CVMFS_FOLLOW_REDIRECTS to its value, by default unset.cvmfs_timeout
cvmfs timeout setting, see params.pp for default.cvmfs_timeout_direct
cvmfs timeout to direct connections, see params.pp for default.cvmfs_nfiles
Number of open files, system setting, see params.pp for default.cvmfs_syslog_level
Default is in params.ppcvmfs_tracefile
Create a tracefile at this location.cvmfs_debuglog
Create a debug log file at this location.cvmfs_max_ttl
Max ttl, see params.pp for default.cvmfs_version
Version of cvmfs to install , default is present.cvmfs_yum
Yum repository URL for cvmfs.cvmfs_yum_priority
Yum priority of repositories, defaults to 80.cvmfs_yum_proxy
http proxy for cvmfs yum package repositorycvmfs_yum_config
Yum repository URL for cvmfs site configs.cvmfs_yum_config_enabled
Defaults to false, set to true to enable.cvmfs_yum_testing
Yum repository URL for cmvfs testing repository.cvmfs_yum_testing_enabled
Defaults to false, should the testing repository be enabled.cvmfs_yum_testsing_enabled
TO DOCcvmfs_yum_gpgcheck
Defaults to true, set to false to disable GPG checking (Do Not Do This)cvmfs_yum_gpgkey
Set a custom GPG key for yum repos, you must deploy it yourself.cvmfs_yum_manage_repo
Defaults to true, set to false to disable yum repositories management.cvmfs_use_geoapi
TO DOCcvmfs_repositories
By default undef andCVMFS_REPOSITORIES
indefault.local
will be populated automatically from what is explicitly mounted withcvmfs::mount
. If this is specified thenCVMFS_REPOSITORIES
list indefault.local
will be exactly managed with this variable. e.gcvmfs-config.cern.ch,atlas.cern.ch
cvmfs_hash
Rather than using cvmfs::mount defined type a hash of mounts can be sepecfied. cvmfs_hash {'myrepo' => {'cvmfs_server_url' => 'http://web.example.org/cvmfs/ams.example.org/}cvmfs_env_variables
$cvmfs_env_variables = {'CMS_LOCAL_SITE' => '' will produceexport CMS_LOCAL_SITE=<path to siteconf>
in the default.local file.
Puppet databindings allows all the above settings to be set via hiera. In
this case it is not nescesary to include class{'cvmfs':}
.
---
cvmfs::cvmfs_quota_limit: 100
cvmfs::cvmfs_nfiles: 20000
Cvmfs::Mount Type
To mount individual repositories optionally with a particular configuration on each repository. e.g
cvmfs::mount{'lhcb.example.org':
}
cvmfs::mount{'atlas.example.org':
cvmfs_timeout => 50
}
cvmfs::mount{'cms.example.org':
cvmfs_timeout => 100,
cvmfs_server_url => 'http://web.example.org/cms.cern.ch'
}
Cvmfs::Mount Type Parameters
namevar
The namevar is the repository name, e.g atlas.example.chcvmfs_repo_list
A boolean defaults totrue
. Should this repository be included in the list of repositories listed asCVMFS_REPOSITORIES
with/etc/cvmfs/default.local
. This is ignored ifcvmfs_repositories
is set on the main class.cvmfs_follow_redirects
Sets CVMFS_FOLLOW_REDIRECTS to its value, by default unset.mount_options
If the mount_method is mount then this specifies the mount options. By default:nodev,_netdev,defaults
.cvmfs_memcache_size
Size of the CernVM-FS meta-data memory cache in Megabyte.cvmfs_repository_tag
Specify a tag , setsCVMFS_REPOSITORY_TAG
- TBC
In addition to creating mounts as above the
create_resources('cvmfs::mount',{})
function is called
allowing the mounts to be specified in a hiera yaml file:
---
cvmfs::cvmfs_hash:
atlas.example.ch:
cvmfs_quota_limit: 10000
atlas-condb.example.ch:
{}
atlas-nightlies.example.ch:
cvmfs_server_url: http://cvmfs-atlas-nightlies.example.ch/cvmfs/atlas-nightlies.example.ch
which will enable these three mount points with the specified options.
Cvmfs::Domain Type
A cvmfs domain file can be created with the cvmfs::domain type
cvmfs::domain{'example.org':
cvmfs_server_url => 'http://host1.example.org/@repo@;http://host2.example2.org/@repo@',
cvmfs_public_key => '/etc/cvmfs/keys/key1.pub,/etc/cvmfs/keys/key2.pub'
}
Cvmfs::Domain Type Parameters
namevar
The namevar is the domain name, e.g example.chcvmfs_follow_redirects
Sets CVMFS_FOLLOW_REDIRECTS to its value, by default unset.- TBC
Fsck Module
An optional class 'cvmfs::fsck' can be included to enable a cron job to regualarly run fsck on cvmfs systems.
class{'cvmfs::fsck':
options => '-p',
onreboot => true
}
In addition a cron will be created to purge quarentine corrupted files after 30 days.
Fsck Options
options
Will pass parameters to thecvmfs_fsck
command, by default none will be passed.onreboot
If set to true a @reboot job will be set to runcvmfs_fsck
at boot time. Default is false.
Stratum 0 Configuration
There are currently two options to configure a stratum 0. The class method only supports one stratum one and will at some point be deprecated.
Stratum 0 Configuration as a Class
class{'cvmfs::server':
repo => 'ilc.example.org',
pubkey => 'public.example.org'
}
See the docs in cvmfs::server for explanation of parameters.
Stratum 0 Configuration as a Defined Type
A new method where each stratum 0 can be configured as an instance. The advantage here is that multiple stratum zeros can be configured per server. The previous class method will be deprecated at some future point.
cvmfs::zero{'files.example.org':
repo_store => '/mybigdisk',
spool_store => '/var/spool/cvmfs',
user => steve,
uid => 500
}
Stratum 0 Parameters
clientuesr
If set will specify the user running the cvmfs_client on the server. Optinal.claim_ownership
. By default false if true it enables theCVMFS_CLAIM_OWNERSHIP
option the server's client instance.group
The group name that will manage the repository and own the files on the server.home
The home directory of theuser
account that owns the cvmfs repositories. The default value isrepo_store
/repo
/user
.gid
The gid fo thegroup
, it defaults to the be same as theuid
, defaults to theuser
setting.repo the fully qualified repository name. Defaults the *name* value of the instance. e.g
example.domain.org`.repo_store
large disk location where the cvmfs repositories are stored. Defaults to/srv/cvmfs
.nofiles
The nofiles theuser
is permitted to open. Defaults to65000
spool_store
location of files internal to a cvmfs server.uid
The uid of theuser
user
The user name that will manage the repository and own the files on the server.auto_tag
Boolean to setCVMFS_AUTO_TAG
, defaults to false.garbage_collection
to setCVMFS_GARBAGE_COLLECTION
defaults to false.auto_gc
to setCVMFS_AUTO_GC
defaults to false.auto_gc_timespan
to setCVMFS_AUTO_GC_TIMESPAN
defaults to3 days ago
.ignore_xdir_hardlinks
boolean to set toCVMFS_IGNORE_XDIR_HARDLINKS
defaults to false.
Stratum 0 Examples
A common case is to mount a device or nfs storage volume to use as the repo
store. In this case the mount should happen before cvmfs::zero populates the
area. For example two repositories stoed on two block device /dev/vdb
and /dev/vdc
.
mount{'/srv/cvmfs/files.example.org':
ensure => mounted,
device => '/dev/vdb',
options => 'rw,noatime,nodiratime,nobarrier,user_xattr'
require => File['/srv/cvmfs/files.example.org'],
before => File['/srv/cvmfs/files.example.org/data']
}
mount{'/srv/cvmfs/objects.example.org':
ensure => mounted,
device => '/dev/vdc',
options => 'rw,noatime,nodiratime,nobarrier,user_xattr'
require => File['/srv/cvmfs/objects.example.org'],
before => File['/srv/cvmfs/objects.example.org/data']
}
cvmfs::zero{'files.example.org':
user => 'steve',
uid => 200,
}
cvmfs::zero{'objects.example.org':
user => 'andrew',
uid => 201,
}
Migrating Stratum 0 Class to Stratum 0 Type
The class based stratum 0 will be deprecated at some point. The differences are:
- The new type no longer attempts to manage a symbolic link to a master key.
If desired this should be done by the addition of symbolic link outside of
cvmfs::zero
. - The new type no longer supports mounting e.g nfs volumes. This myst be done externally as per the example above.
Replacing
class{'cvmfs::server':
repo => 'ilc.example.org',
pubkey => 'public.example.org',
user => 'shared',
uid => 101
}
with
cvmfs::zero{'ilc.example.org':
user => shared,
uid => 101
}
file{'/etc/cvmfs/keys/ilc.example.org':
ensure => link,
target => '/etc/cvmfs/keys/public.example.org.pem',
require => Package['cvmfs']
}
Stratum One Configuration
A stratum one can be configured for multiple repositories with a
cvmfs::one{'mice.example.org':
origin => 'http://cvmfs01.example.org/cvmfs',
keys => ['/etc/cvmfs/keys/example1.pub','/etc/cvmfs/keys/example1.pub'],
mime_expire => 61,
}
See cvmfs::one.pp for more details of parameters.
Tests
To run standalone tests
bundle install
bundle exec rake validate
bundle exec rake lint
bundle exec rake spec
The acceptance tests by default use docker ensure that is working or provide beaker configuration for your own hypervisor.
bundle install
BEAKER_debug=yes BEAKER_set=centos-7-x86_64-docker bundle exec rspec spec/acceptance
License
Apache II License for all files except automaster.aug which is copied from the http://augeas.net project. The automaster.aug file is LGPL v2+.
Contact
Steve Traylen steve.traylen@cern.ch
Support
2020-08-13 - Release 7.3.0
- New
cvmfs_repositories
parameter to exactly specifyCVMFS_REPOSITORIES
indefault.local
as used bycvmfs_config probe
for instance.
2020-08-11 - Release 7.2.0
- New
cvmfs_repository_tag
parameter to mount command.
2020-06-34 - Release 7.1.0
- New pararmeter
cvmfs_instrument_fuse
parameter. - Enable cmvfs-fsck.timer so it starts on boot.
- Remove legacy limits module requirement.
2020-02-10 - Release 7.0.1
- Bugfix on CentOS 8 only. Files in
/etc/autofs.master.d/
must end in .autofs to be included.
2020-01-06 - Release 7.0.0
- remove SLC5 support
- Add CentOS 8 support
- fsck cron jobs are now a systemd timer on 8
- A file is placed in auto.master.d rather than editing auto.master
- New parameters
cvmfs_shared_cache
andcvmfs_alien_cache
- Docs - correct cvmfs mount hash example
cvmfs_yum_includepkgs
now accepts an array of packages. Setting it as a string of packages is deprecated.
2019-03-14 - Release 6.2.0
- Support cvmfs_http_proxy on a mount
2019-01-28 - Release 6.1.0
- Support EXTERNAL repository parameters
2018-12-03 - Release 6.0.1
- Rely on $PATH to find
nice
command in fsck cron.
2018-12-03 - Release 6.0.0
- New parameters
cvmfs_dns_min_ttl
andcvmfs_dns_max_ttl
. Unit is in seconds, support added incvmfs-2.5.2
- New parameter
cvmfs_yum_priority
to set yum repo priority. The default is the existing value so no change. - Fixtures for Puppet6 support.
- All the deprecated explicit hiera calls that were issuing notify calls for the last year have been completely removed.
- Puppet 4 types have now been added and much stronger
type checking of input is now done. e.g while a
cvmfs_quota_limit
of'1234'
or1234
used to be permitted only the latter is now permitted.
2018-06-28 - Release 5.2.0
- New stratum one configuration
mime_expire
defaulting 61 seconds.
2018-06-28 - Release 5.1.0
cvmfs_force_singing
parameter removed. Not used by cvmfs for years.- The selinux context for the cache is now correctly set to
cvmfs_cache_t
- The tmpclean of quarantine directory only runs if quarantine directory exists.
- Fix paths to awk, nice.
- Support debian based distributions for the first time.
2018-01-24 - Release 5.0.0
-
Deprecates the following explicit hiera variables:
cvmfs_quota_limit
,cvmfs_quota_ratio
,cvmfs_http_proxy
,cvmfs_server_url
,cvmfs_cache_base
,cvmfs_timeout
,cvmfs_timeout_direct
,cvmfs_nfiles
,cvmfs_public_key
,cvmfs_force_signing
,cvmfs_syslog_level
,cvmfs_tracefile
,cvmfs_debuglog
,cvmfs_max_ttl
,cvmfs::mount
,cvmfs_yum
,cvmfs_yum_testing
,cvmfs_yum_testing_enabled
,cvmfs_kernel_version
,cvmfs_yum_kernel
,cvmfs_yum_kernel_enabled
,cvmfs_sync_minute
.In almost every case the existing conventional class parameter can be used instead. e.g
cvmfs_quota_limit
becomescvmfs::cvmfs_quota_limit
. The class parameter can be specified via hiera if so wished. These parameters have been available, documented and advised for sometime.Any use of the above explicit hiera lookups will work for now but a
notify
will be issued offering advice on what action to take.The next version
puppet-cvmfs
will completely drop thier support as well as support for puppet3.
2017-05-18 - Release 4.2.0
- New paramter
mime_expire
tocvmfs::zero
to specify expirey headers for cvmfs files in apache. Defaults to120
. Unit is seconds. - New optional parameter
cvmfs_claim_ownership
to main class and to mount type. - Bugfixes for latest concat modules.
2017-04-04 - Release 4.1.0
- New parameter
cvmfs_memcache_size
to main class and to mount type.
2017-02-15 - Release 4.0.0
- cvmfs::server is now deprecated and will be removed at next release. README.md contains details for migating to cvmfs::zero defined type.
2017-02-08 - Release 3.2.0
- Updates for cvmfs_server version 2.3
2016-09-14 - Release 3.1.0
- Wrap autofs service with ensure_resource to make co-existing easier.
2016-08-18 - Release 3.0.1
- Bug fix fact for ruby 1.8
2016-08-18 - Release 3.0.0
- DEPRCATION: Variable
cvmfs::config_automounter
is now deprecated. If it is set to the non defaultfalse
compilation will fail. - New variable
cvmfs::mount_method
is a string defaulting toautofs
and provides the current default behaviour to use automounter configuration. It can also be set tomount
in which will use the puppet mount type which currently mounts and adds an/etc/fstab
entry. It can also be set tonone
in which case no mount configuration will be applied. - Fixes #56 tmpwatch is now installed if fsck crons are enabled.
2016-04-29 - Release 2.0.0
- Remove all support for cvmfs 2.0.
- Only one rather than two puppet runs are now requied.
- Drop bundled augeas automaster lens.
- Stratum 1 apache file update for cvmfs 2.2.
2016-03-07 - Release 1.0.3
- Stratum 1 cron jobs more frequent.
2016-03-07 - Release 1.0.2
- Puppetforge publish problem
2016-03-04 - Release 1.0.0
- New parameter
cvmfs::cvmfs_yum_manage_repo
defaults to true allows all yum managment to be disabled. Baptiste Grenier - Package installations now wrapped in ensure_package to ease duplicate declarations Pat Riehecky
- Stratum 0 only, new parameter
cvmfs::zero::creator_version
defaults to2.1.19
. - Stratum 0 only, new json mime types to support cvmfs 2.2.
- Stratum 0 only, rdonly directory no longer mounted at boot time.
- Stratum 0 only, backwards incompatible change, the gid and group name now MUST be specified.
- Fixes for puppet 4 support.
2015-08-04 - Release 0.9.0
- A tmpwatch -f 30d now runs as weekly cron withing cvmfs::fsck to purge quarentined files.
2015-07-07 - Release 0.8.0
Features
- Tests now check default configuration do not change.
- CVMFS_FOLLOW_REDIRECTS can now be set globally, per repo or per domain.
- New onreboot paramter to fsck module runs an fsck on reboot. Default false.
- zfs kernel module built for aufs kernel may now be installed on stratum0s.
2015-06-22 - Release 0.7.0
Features
- New option defaults cvmfs::fsck::options allows an option, e.g. -p to parsed to cvmfs_fsck cron.
Bugfixes
- Reintroduce $::concat_basedir in tests since puppetlabs-concat was reverted to version 1.
2015-06-10 - Release 0.6.0
Features
- Garbage collection parameters for stratum 0.
- CVMFS_IGNORE_XDIR_HARDLINKS can be set on stratum 0.
##2015-05-06 - Release 0.5.0
Deprecations
- All the hiera variables explicity called from params.pp
file are now deprecated. These variables include.
cvmfs_config_automaster
,cvmfs_quota_limit
,cvmfs_quota_limit
,cvmfs_quota_ratio
,cvmfs_http_proxy
,cvmfs_server_url
,cvmfs_cache_base
,cvmfs_timeout
,cvmfs_nfiles
,cvmfs_public_key
,cvmfs_force_signing
,cvmfs_syslog_level
,cvmfs_tracefile
,cvmfs_debuglog
cvmfs_max_ttl
,cvmfs::mount
,cvmfsversion
,cvmfs_yum
,cvmfs_yum_testing
,cvmfs_yum_proxy
cvmfs_kernel_version
,cvmfs_yum_kernel
,cvmfs_yum_kernel_enabled
Instead use the hiera binding that maps to one of the paramters of thecvmfs
class, e.gcvmfs_quota_limit
becomescvmfs::cvmfs_quota_limit
A future version of this module will print a warning if these variables are still being used.
Features
- Beaker acceptence tests now added for client. Configures a client and mounts the cms.cern.ch and atlas.cern.ch repositories.
- The fact ::operatingsystemmajrelease is used everywhere now to determine major OS version.
- Unit tests for yumrepos.
- The GPG key location for packages in the cvmfs yum repositories can now be specified.
- The GPG check of packages can be disabled in the yum repository configuration.
- The new cvmfs-config repository is now configured but is disabled by default.
Bugfixes
- The examples in the README now work.
2015-04-30 - Release 0.4.4
Bugfixes
- Correct erwbgy-limits version dependency.
##2015-04-30 - Release 0.4.3
Bugfixes
- Fix github to puppetforge publishing.
2015-04-30 - Release 0.4.2
Bugfixes
- Fix github to puppetforge publishing.
2015-04-29 - Release 0.4.1
Bugfixes
- Fix github to puppetforge publishing.
2015-04-29 - Release 0.4.0
Features
- New paramter
cvmfs_repo_list
allows entries of CVMFS_REPOSITORIES to be fine tuned per cvmfs::mount. https://github.com/cvmfs/puppet-cvmfs/pull/22 - New paramter
manage_autofs_service
controls if autofs service should be managed module. Existing paramterconfig_automaster
now only configuresauto.master file
- @jcpunk. - New type cvmfs::zero to configure a CvmFS stratum 0. The existing class cvmfs::server for stratum 0s will be deprecated at some future date.
- Doc changes for latest PL style guide. CHANGELOG is now markdown. Docs for classes and types being migrated from them to README.md.
- puppet-cvmfs is now autodeployed to puppetforge once tagged.
- CvmFS env settings can now be set globally as well per domain or mount.
- New variable
cvmfs_domain_hash
allows a hash of CvmFS domains to loaded from hiera. - New variable
cvmfs_use_geoapi
can be set globally, per domain or mount to influenceCVMFS_USE_GEOAPI
- rspec functional tests now exist for class cvmfs and types cvmfs::zero, cvmfs::mount and cvmfs::domain.
Bugfixes
- cvmfs-config* packages can now be installed on cvmfs servers.
- All references to deprecated concat::setup removed - @berghaus.
- Vague use of undef was failing under furture parser or puppet 4.0
2015-03-06 - Release 0.3.3
Features
- do not control autofs service when asked not to.
- Install mod_wsgi and configure for geoip on stratum 1.
- Allow a http proxy for yum repos to be configured.
- Migrate to puppetlabs lint tests.
Bugfixes
2014-06-18 - Release 0.3.2
Features
- server - seperate yum configuration for server and client.
- server - Add
nofiles
paramter to server class to allow no open files to be specified for the user running the cvmfs server. - server - An exact kernel version is no longer specified.
- README file converted to markdown, now README.md.
- client - new defined type cvmfs::domain to create .local file in /etc/cvmfs/domain.d.
- client - cvmfs::init class is now paramatised so supports that in addition to hiera.
Bugfixes
- server - force a public key file location to be specified.
- server - when disabling selinux on server specify an augeas scope
2014-03-27 - Release 0.2.2
Features
Bugfixes
- bugfix for cvmfs_public_key variable.
2014-01-16 - Release 0.2.1
Features
Bugfixes
- Puppet 3.4 compatability for ensure_resources.
Dependencies
- puppetlabs/concat (>=1.0.0 < 6.0.0)
- puppetlabs/stdlib (>=4.13.1 < 6.0.0)
- puppetlabs/firewall (>=1.0.0 < 2.0.0)