Documentation
Table of Contents
- Overview
- This is a SIMP module
- Module Description
- Terminology
- Usage
- File Store and Plugin
- Limitations
- Plugin Development
- simpkv Development
Overview
This is a SIMP module
This module is a component of the System Integrity Management Platform, a compliance-management framework built on Puppet.
If you find any issues, please submit them via JIRA.
Please read our Contribution Guide.
Module Description
Provides an abstract library that allows Puppet to access one or more key/value stores.
This module provides
-
a standard Puppet language API (functions) for using key/value stores
- The API is modeled after https://github.com/docker/libkv#interface.
- See REFERENCE.md for more details on the available functions.
-
a configuration scheme that allows users to specify per-application use of different key/value store instances
-
adapter software that loads and uses store-specific interface software provided by the simpkv module itself and other modules
-
a Ruby API for the store interface software that developers can implement to provide their own store interface
-
a file-based store on the local filesystem and its interface software.
- Future versions of this module will provide a distributed key/value store.
If you find any issues, they may be submitted to our bug tracker.
Terminology
The following terminology will be used throughout this document:
- backend - A specific key/value store, e.g., files on a local filesystem, Consul, Etcd, Zookeeper.
- plugin - Ruby software that interfaces with a specific backend to affect the operations requested in simpkv Puppet functions.
- plugin instance - Instance of the plugin that handles a unique backend configuration.
- plugin adapter - Ruby software that loads, selects, and executes the appropriate plugin software for a simpkv function call.
Usage
Using simpkv
is simple:
-
Use
simpkv
functions to store and retrieve key/value pairs in your Puppet code. -
Configure the backend(s) to use in Hieradata.
-
Reconfigure the backend(s) in Hieradata, as your needs change.
- No changes to your Puppet code will be required.
- Just transfer your data from the old key/value store to the new one.
The backend configuration of simpkv
can be as simple as you want (one backend)
or complex (multiple backends servicing different applications). Examples of
both scenarios will be shown in this section, along with a configuration
reference.
Single Backend Example
This example will store and retrieve host information using simpkv function signatures that assume the default backend and hieradata that only configures the default backend.
To store a node's hostname and IP address:
simpkv::put("hosts/${facts['clientcert']}", $facts['ipaddress'])
To create a hosts file using the list of stored host information:
$result = simpkv::list('hosts')
$result['keys'].each |$host, $info | {
host { $host:
ip => $info['value'],
}
}
In hieradata, configure the default backend in the simpkv::options
Hash. This
example, will configure simpkv's file backend.
simpkv::options:
# Hash of backend configurations.
# - We have only the required 'default' entry which will apply to
# all simpkv calls.
backends:
default:
# This is the advertised type for simpkv's file plugin.
type: file
# This is a unique id for this configuration of the 'file' plugin.
id: file
# plugin-specific configuration
root_path: "/var/simp/simpkv/file"
lock_timeout_seconds: 30
Multiple Backends Example
This example will store and retrieve host information using simpkv function signatures that request a backend based on an application id and multi-backend hieradata that supports the request. The function signatures and hieradata are a little more complicated, but still relatively straightforward to understand.
To store a node's hostname and IP address using the backend servicing myapp1
:
$simpkv_options = { 'app_id' => 'myapp1' }
$empty_metadata = {}
simpkv::put("hosts/${facts['clientcert']}", $facts['ipaddress'], $empty_metadata, $simpkv_options)
To create a hosts file using the list of stored host information using the
backend servicing myapp1
:
$simpkv_options = { 'app_id' => 'myapp1' }
$result = simpkv::list('hosts', $simpkv_options)
$result['keys'].each |$host, $info | {
host { $host:
ip => $info['value'],
}
}
In hieradata, configure multiple backends in the simpkv::options
Hash.
This example will configure multiple instances of simpkv's file backend.
# The backend configurations here will be inserted into simpkv::options
# below via the alias function.
simpkv::backend::file_default:
type: file
id: default
root_path: "/var/simp/simpkv/file"
simpkv::backend::file_myapp:
type: file
id: myapp
root_path: "/path/to/myapp"
simpkv::backend::file_yourapp:
type: file
id: yourapp
root_path: "/path/to/yourapp"
simpkv::options:
# Hash of backend configurations.
# * Includes application-specific backends and the required default backend.
# * simpkv will use the appropriate backend for each simpkv function call.
backends:
# backend for specific myapp application
"myapp_special_snowflake": "%{alias('simpkv::backend::file_default')}"
# backend for remaining myapp* applications, including myapp1
"myapp": "%{alias('simpkv::backend::file_myapp')}"
# backend for all yourapp* applications
"yourapp": "%{alias('simpkv::backend::file_yourapp')}"
# required default backend
"default": "%{alias('simpkv::backend::file_default')}"
In this example, we are setting the application identifier to myapp1
in
our simpkv function calls. simpkv selects myapp
as the backend to use for
myapp1
using the following simple search algorithm:
- First, it looks for a backend named for the application id.
- Next, it looks for the longest backend name matching the start of the application id.
- Finally, if no match is found, it defaults to a backend named
default
.
Binary Value Example
simpkv is able to store and retrieve binary values, provided the Puppet code uses the appropriate configuration and functions/types for binary data.
- IMPORTANT: Puppet 5 does not fully support binary data. So,
although simpkv will properly serialize and deserialize the data, the
binary data can only be used for Puppet
file
resources when applied withpuppet apply
, notpuppet agent
.
Below is an example of using simpkv for a binary value.
To store the content of a generated keytab file:
# Load in the binary content from a file. Returns a Binary Puppet type.
$original_binary_content = binary_file('/path/to/keytabs/app.keytab')
# Set a key/value pair with the binary content
simpkv::put('app/keytab', $original_binary_content)
To retrieve the keytab binary content and use it in a file
resource:
# Retrieve a binary value from a key/value store and set a Binary variable
$retrieved_result = simpkv::get('app/keytab')
$retrieved_binary_content = Binary.new($retrieved_result['value'], '%r')
# Persist binary data to another file
file { '/different/path/to/keytabs/app.keytab':
content => $retrieved_binary_content
}
Auto-Default Backend
simpkv is intended to be configured via simpkv::options
and any
application-specific configuration passed to the simpkv Puppet functions.
However, to facilitate rollout of simpkv capabilities, (specifically
use of simpkv internally in simplib::passgen
), when simpkv::options
is not set in hieradata, simpkv will automatically use the simpkv file store with
the configuration that is equivalent to the following hieradata:
simpkv::options:
environment: "%{server_facts.environment}"
softfail: false
backend: default
backends:
default:
type: file
id: auto_default
Backend Folder Layout
The storage in a simpkv backend can be notionally represented as a folder tree with key files at terminal nodes. simpkv automatically sets up the folder layout at the top level and the user specifies key files below that. Specifically,
-
simpkv stores global keys directly off the root folder.
-
simpkv stores all other keys in a sub-folder named for the Puppet environment in which the key was created.
-
Further sub-folder trees are allowed for the global or environment-specific keys.
- A relative paths in a key name indicates a sub-folder tree.
-
The actual representation of the root folder is backend specific.
- For the 'file' backend, the root folder is a directory on the local file system of the Puppet server.
Below is an example of a folder tree for the file
backend configured
with an id
of default
:
/var/simp/simpkv/file/default
│
├── app1/ ............. Folder for 'app1' global keys
│ └── global_keyQ ... simpkv::put('app1/global_keyQ', { 'env'=>'' }) in any environment
│
├── dev/ .............. Folder for 'dev' Puppet environment keys
│ └── app1/
│ └── keyA ...... simpkv::put('app1/keyA') in a 'dev' node
│
├── production/ ....... Folder for 'production' Puppet environment keys
│ ├── app1/
│ │ └── keyA ...... simpkv::put('app1/keyA') in a 'production' node
│ ├── app2/
│ │ ├── groupX/
│ │ │ └── keyB
│ │ └── groupY/
│ │ └── keyC .. simpkv::put('app2/groupY/keyC') in a 'production' node
│ └── keyD .......... simpkv::put('keyD') in a 'production' node
│
└── global_keyR ....... simpkv::put('global_keyR', { 'env'=>'' }) in any environment.
simpkv Configuration Reference
The simpkv configuration used for each simpkv function call is comprised of
a merge of a function-provided options Hash, Hiera configuration specified
by the simpkv::options
Hash, and global configuration defaults. The merge
is executed in a fashion to ensure the function-provided options take
precedence over the simpkv::options
Hiera values.
The merged simpkv configuration contains global and backend-specific configurations, along with an optional application identifier. The primary keys in this Hash are as follows:
-
app_id
: Optional String in simpkv function calls, only. Specifies an application name that can be used to identify which backend configuration to use via fuzzy name matching, in the absence of thebackend
option. (See Backend Selection).- More flexible option than
backend
. - Useful for grouping together simpkv function calls found in different catalog resources.
- More flexible option than
-
backend
: Optional String. Specifies a definitive backend configuration to use.-
Takes precedence over
app_id
. -
When present, must match a key in
backends
and will be used unequivocally.- If that backend does not exist in
backends
, the simpkv function will fail.
- If that backend does not exist in
-
When absent, the backend configuration will be selected from the set of entries in
backends
, using theapp_id
option if specified. (See Backend Selection).
-
-
backends
: Required Hash. Specifies backend configurations. Each key is the name of a backend configuration and its value contains the corresponding configuration Hash.- Each key is a String.
- Must include a 'default' key.
- More than one key can use the same backend configuration.
- See Backend Configuration Entries for more details about a backend configuration Hash.
-
environment
: Optional String. Puppet environment to prepend to keys.- When set to a non-empty string, it is prepended to the key or key folder used in a backend operation.
- Should only be set to an empty string when the key being accessed is truly global.
- Defaults to the Puppet environment for the node when absent.
-
softfail
: Optional Boolean. Whether to ignore simpkv operation failures.- When
true
, each simpkv function will return a result object even when the operation failed at the backend. - When
false
, each simpkv function will fail when the backend operation failed. - Defaults to
false
when absent.
- When
Backend Configuration Entries
Each backend configuration entry in backends
is a Hash. The Hash must
contain type
and id
keys, where the (type
,id
) pair defines a unique
configuration.
type
must be unique across all backend plugins, including those provided by other modules.id
must be unique for a each distinct configuration for atype
.- Other keys for configuration specific to the backend may also be present.
Backend Selection
The backend to use for a simpkv Puppet function call will be determined from the merged simpkv options Hash as follows:
-
If a specific backend is requested via the
backend
key in the merged simpkv options Hash, that backend will be selected.- If that backend does not exist in
backends
, the simpkv function will fail.
- If that backend does not exist in
-
Otherwise, if an
app_id
option is specified in the merged simpkv options Hash and it matches a key in thebackends
Hash, exactly, that backend will be selected. -
Otherwise, if an
app_id
option is specified in the merged simpkv options Hash and it starts with the key in thebackends
Hash, that backend will be selected.- When multiple backends satisfy the 'start with' match, the backend with the most matching characters is selected.
-
Otherwise, if the
app_id
option does not match any key in in thebackends
Hash or is not present, thedefault
backend will be selected.
File Store and Plugin
simpkv provides a file-based key/value store and its plugin. This file store
maintains individual key files on a local filesystem, has a backend type file
,
and supports the following plugin-specific configuration parameters.
-
root_path
: Root directory path for the key files- Defaults to
/var/simp/simpkv/file/<id>
when that directory can be created or '<Puppet[:vardir]>/simp/simpkv/' otherwise.
- Defaults to
-
lock_timeout_seconds
: Maximum number of seconds to wait for an exclusive file lock on a file modifying operation before failing the operation.- Defaults to 5 seconds.
Limitations
-
SIMP Puppet modules are generally intended to be used on a Red Hat Enterprise Linux-compatible distribution such as EL6 and EL7.
-
simpkv's file plugin is only guaranteed to work on local filesystems. It may not work on shared filesystems, such as NFS.
-
simpkv
only supports the use of binary data for the value when that data is a PuppetBinary
. It does not support binary data which is a sub-element of a more complex value type (e.g.Array[Binary]
orHash
that has a key or value that is aBinary
).
Plugin Development
Plugin Loading
Each plugin (store interface) is written in pure Ruby and, to prevent cross-environment contamination, is implemented as an anonymous class that is automatically loaded by the simpkv adapter with each Puppet compile. You do not have to do anything special to have your plugin loaded, provided you follow the instructions in the next section.
Implementing the Store Interface API
To create your own plugin
- Create a
lib/puppet_x/simpkv
directory within your store plugin module. - Copy
lib/puppet_x/simpkv/plugin_template.rb
from the simpkv module into that directory with a name<your plugin name>_plugin.rb
. For example,nfs_file_plugin.rb
. - READ all the documentation in your plugin skeleton, paying close attention
the
IMPORTANT NOTES
discussion. - Implement the body of each method as identified by a
FIXME
. Be sure to conform to the API for the method. - Write unit tests for your plugin, using the unit tests for simpkv's file
plugin,
spec/unit/puppet_x/simpkv/file_plugin_spec.rb
as an example. That test shows you how to instantiate an object of your plugin for testing purposes. - Write acceptance tests for your plugin, using the acceptance tests for
simpkv's file plugin,
spec/acceptances/suites/default/file_plugin_spec.rb
, as an example. That test uses a test module,spec/support/simpkv_test
to exercise the the simpkv API and verify its operation. - Document your plugin's type and configuration parameters in the README.md for your store plugin module.
simpkv Development
Please read our [Contribution Guide] (https://simp.readthedocs.io/en/stable/contributors_guide/index.html).
Unit tests
Unit tests, written in rspec-puppet
can be run by calling:
bundle exec rake spec
Acceptance tests
To run the system tests, you need Vagrant installed. Then, run:
bundle exec rake beaker:suites
Some environment variables may be useful:
BEAKER_debug=true
BEAKER_provision=no
BEAKER_destroy=no
BEAKER_use_fixtures_dir_for_modules=yes
BEAKER_debug
: show the commands being run on the STU and their output.BEAKER_destroy=no
: prevent the machine destruction after the tests finish so you can inspect the state.BEAKER_provision=no
: prevent the machine from being recreated. This can save a lot of time while you're writing the tests.BEAKER_use_fixtures_dir_for_modules=yes
: cause all module dependencies to be loaded from thespec/fixtures/modules
directory, based on the contents of.fixtures.yml
. The contents of this directory are usually populated bybundle exec rake spec_prep
. This can be used to run acceptance tests to run on isolated networks.
Reference
Table of Contents
Functions
simpkv::delete
: Deletes akey
from the configured backend.simpkv::deletetree
: Deletes a whole folder from the configured backend.simpkv::exists
: Returns whether key or key folder exists in the configured backend.simpkv::get
: Retrieves the value and any metadata stored atkey
from the configured backend.simpkv::list
: Returns a listing of all keys and sub-folders in a folder. The list operation does not recurse through any sub-folders. Only information abosimpkv::put
: Sets the data atkey
to the specifiedvalue
in the configured backend. Optionally sets metadata along with thevalue
.simpkv::support::config::merge
: Create merged backend configuration and then validate it. The merge entails the following operations: * Theoptions
argument is merged witsimpkv::support::config::validate
: Validate backend configurationsimpkv::support::key::validate
: Validates key conforms to the simpkv key specification simpkv key specification Key must contain only the following characters:simpkv::support::load
: Load simpkv adapter and plugins and add simpkv 'extension' to the catalog instance, if it is not present
Functions
simpkv::delete
Type: Ruby 4.x API
Deletes a key
from the configured backend.
Examples
Delete a key using the default backend
simpkv::delete("hosts/${facts['fqdn']}")
Delete a key using the backend servicing an application id
simpkv::delete("hosts/${facts['fqdn']}", { 'app_id' => 'myapp' })
simpkv::delete(String[1] $key, Optional[Hash] $options)
Deletes a key
from the configured backend.
Returns: Boolean
true
when backend operation succeeds; false
when the
backend operation fails and 'softfail' is true
in the merged backend
options
Raises:
ArgumentError
If the key or merged backend config is invalidLoadError
If the simpkv adapter cannot be loadedRuntimeError
If the backend operation fails, unless 'softfail' istrue
in the merged backend options.
Examples
Delete a key using the default backend
simpkv::delete("hosts/${facts['fqdn']}")
Delete a key using the backend servicing an application id
simpkv::delete("hosts/${facts['fqdn']}", { 'app_id' => 'myapp' })
key
Data type: String[1]
The key to remove. Must conform to the following:
-
Key must contain only the following characters:
- a-z
- A-Z
- 0-9
- The following special characters:
._:-/
-
Key may not contain '/./' or '/../' sequences.
options
Data type: Optional[Hash]
simpkv configuration that will be merged with
simpkv::options
. All keys are optional.
Options:
-
'app_id'
String
: Specifies an application name that can be used to identify which backend configuration to use via fuzzy name matching, in the absence of thebackend
option.- More flexible option than
backend
. - Useful for grouping together simpkv function calls found in different catalog resources.
- When specified and the
backend
option is absent, the backend will be selected preferring a backend in the mergedbackends
option whose name exactly matches theapp_id
, followed by the longest backend name that matches the beginning of theapp_id
, followed by thedefault
backend. - When absent and the
backend
option is also absent, this function will use thedefault
backend.
- More flexible option than
-
'backend'
String
: Definitive name of the backend to use.- Takes precedence over
app_id
. - When present, must match a key in the
backends
option of the merged options Hash or the function will fail. - When absent in the merged options, this function will select
the backend as described in the
app_id
option.
- Takes precedence over
-
'backends'
Hash
: Hash of backend configurations-
Each backend configuration in the merged options Hash must be a Hash that has the following keys:
type
: Backend type.id
: Unique name for the instance of the backend. (Same backend type can be configured differently).
-
Other keys for configuration specific to the backend may also be present.
-
-
'environment'
String
: Puppet environment to prepend to keys.- When set to a non-empty string, it is prepended to the key used in the backend operation.
- Should only be set to an empty string when the key being accessed is truly global.
- Defaults to the Puppet environment for the node.
-
'softfail'
Boolean
: Whether to ignore simpkv operation failures.- When
true
, this function will return a result even when the operation failed at the backend. - When
false
, this function will fail when the backend operation failed. - Defaults to
false
.
- When
simpkv::deletetree
Type: Ruby 4.x API
Deletes a whole folder from the configured backend.
Examples
Delete a key folder using the default backend
simpkv::deletetree("hosts")
Delete a key folder using the backend servicing an appliction id
simpkv::deletetree("hosts", { 'app_id' => 'myapp' })
simpkv::deletetree(String[1] $keydir, Optional[Hash] $options)
Deletes a whole folder from the configured backend.
Returns: Boolean
true
when backend operation succeeds; false
when the
backend operation fails and 'softfail' is true
in the merged backend
options
Raises:
ArgumentError
If the key folder or merged backend config is invalidLoadError
If the simpkv adapter cannot be loadedRuntimeError
If the backend operation fails, unless 'softfail' istrue
in the merged backend options.
Examples
Delete a key folder using the default backend
simpkv::deletetree("hosts")
Delete a key folder using the backend servicing an appliction id
simpkv::deletetree("hosts", { 'app_id' => 'myapp' })
keydir
Data type: String[1]
The key folder to remove. Must conform to the following:
-
Key folder must contain only the following characters:
- a-z
- A-Z
- 0-9
- The following special characters:
._:-/
-
Key folder may not contain '/./' or '/../' sequences.
options
Data type: Optional[Hash]
simpkv configuration that will be merged with
simpkv::options
. All keys are optional.
Options:
-
'app_id'
String
: Specifies an application name that can be used to identify which backend configuration to use via fuzzy name matching, in the absence of thebackend
option.- More flexible option than
backend
. - Useful for grouping together simpkv function calls found in different catalog resources.
- When specified and the
backend
option is absent, the backend will be selected preferring a backend in the mergedbackends
option whose name exactly matches theapp_id
, followed by the longest backend name that matches the beginning of theapp_id
, followed by thedefault
backend. - When absent and the
backend
option is also absent, this function will use thedefault
backend.
- More flexible option than
-
'backend'
String
: Definitive name of the backend to use.- Takes precedence over
app_id
. - When present, must match a key in the
backends
option of the merged options Hash or the function will fail. - When absent in the merged options, this function will select
the backend as described in the
app_id
option.
- Takes precedence over
-
'backends'
Hash
: Hash of backend configurations-
Each backend configuration in the merged options Hash must be a Hash that has the following keys:
type
: Backend type.id
: Unique name for the instance of the backend. (Same backend type can be configured differently).
-
Other keys for configuration specific to the backend may also be present.
-
-
'environment'
String
: Puppet environment to prepend to keys.- When set to a non-empty string, it is prepended to the key used in the backend operation.
- Should only be set to an empty string when the key being accessed is truly global.
- Defaults to the Puppet environment for the node.
-
'softfail'
Boolean
: Whether to ignore simpkv operation failures.- When
true
, this function will return a result even when the operation failed at the backend. - When
false
, this function will fail when the backend operation failed. - Defaults to
false
.
- When
simpkv::exists
Type: Ruby 4.x API
Returns whether key or key folder exists in the configured backend.
Examples
Check for the existence of a key in the default backend
if simpkv::exists("hosts/${facts['fqdn']}") {
notify { "hosts/${facts['fqdn']} exists": }
}
Check for the existence of a key in the backend servicing an application id
if simpkv::exists("hosts/${facts['fqdn']}", { 'app_id' => 'myapp' }) {
notify { "hosts/${facts['fqdn']} exists": }
}
Check for the existence of a key folder in the default backend
if simpkv::exists("hosts") {
notify { 'hosts folder exists': }
}
simpkv::exists(String[1] $key, Optional[Hash] $options)
Returns whether key or key folder exists in the configured backend.
Returns: Enum[Boolean,Undef]
If the backend operation succeeds, returns
true
or false
; if the backend operation fails and 'softfail' is true
in the merged backend options, returns nil
Raises:
ArgumentError
If the key or merged backend config is invalidLoadError
If the simpkv adapter cannot be loadedRuntimeError
If the backend operation fails, unless 'softfail' istrue
in the merged backend options.
Examples
Check for the existence of a key in the default backend
if simpkv::exists("hosts/${facts['fqdn']}") {
notify { "hosts/${facts['fqdn']} exists": }
}
Check for the existence of a key in the backend servicing an application id
if simpkv::exists("hosts/${facts['fqdn']}", { 'app_id' => 'myapp' }) {
notify { "hosts/${facts['fqdn']} exists": }
}
Check for the existence of a key folder in the default backend
if simpkv::exists("hosts") {
notify { 'hosts folder exists': }
}
key
Data type: String[1]
The key or key folder to check. Must conform to the following:
-
Key must contain only the following characters:
- a-z
- A-Z
- 0-9
- The following special characters:
._:-/
-
Key may not contain '/./' or '/../' sequences.
options
Data type: Optional[Hash]
simpkv configuration that will be merged with
simpkv::options
. All keys are optional.
Options:
-
'app_id'
String
: Specifies an application name that can be used to identify which backend configuration to use via fuzzy name matching, in the absence of thebackend
option.- More flexible option than
backend
. - Useful for grouping together simpkv function calls found in different catalog resources.
- When specified and the
backend
option is absent, the backend will be selected preferring a backend in the mergedbackends
option whose name exactly matches theapp_id
, followed by the longest backend name that matches the beginning of theapp_id
, followed by thedefault
backend. - When absent and the
backend
option is also absent, this function will use thedefault
backend.
- More flexible option than
-
'backend'
String
: Definitive name of the backend to use.- Takes precedence over
app_id
. - When present, must match a key in the
backends
option of the merged options Hash or the function will fail. - When absent in the merged options, this function will select
the backend as described in the
app_id
option.
- Takes precedence over
-
'backends'
Hash
: Hash of backend configurations-
Each backend configuration in the merged options Hash must be a Hash that has the following keys:
type
: Backend type.id
: Unique name for the instance of the backend. (Same backend type can be configured differently).
-
Other keys for configuration specific to the backend may also be present.
-
-
'environment'
String
: Puppet environment to prepend to keys.- When set to a non-empty string, it is prepended to the key used in the backend operation.
- Should only be set to an empty string when the key being accessed is truly global.
- Defaults to the Puppet environment for the node.
-
'softfail'
Boolean
: Whether to ignore simpkv operation failures.- When
true
, this function will return a result even when the operation failed at the backend. - When
false
, this function will fail when the backend operation failed. - Defaults to
false
.
- When
simpkv::get
Type: Ruby 4.x API
Retrieves the value and any metadata stored at key
from the
configured backend.
Examples
Retrieve the value and any metadata for a key in the default backend
$result = simpkv::get("database/${facts['fqdn']}")
class { 'wordpress':
db_host => $result['value']
}
Retrieve the value and any metadata for a key in the backend servicing an application id
$result = simpkv::get("database/${facts['fqdn']}", { 'app_id' => 'myapp' })
class { 'wordpress':
db_host => $result['value']
}
simpkv::get(String[1] $key, Optional[Hash] $options)
Retrieves the value and any metadata stored at key
from the
configured backend.
Returns: Enum[Hash,Undef]
Hash containing the value and any metadata upon
success; Undef when the backend operation fails and 'softfail' is true
in the merged backend options
- Hash will have a 'value' key containing the retrieved value
- Hash may have a 'metadata' key containing a Hash with any metadata for the key
Raises:
ArgumentError
If the key or merged backend config is invalidLoadError
If the simpkv adapter cannot be loadedRuntimeError
If the backend operation fails, unless 'softfail' istrue
in the merged backend options.
Examples
Retrieve the value and any metadata for a key in the default backend
$result = simpkv::get("database/${facts['fqdn']}")
class { 'wordpress':
db_host => $result['value']
}
Retrieve the value and any metadata for a key in the backend servicing an application id
$result = simpkv::get("database/${facts['fqdn']}", { 'app_id' => 'myapp' })
class { 'wordpress':
db_host => $result['value']
}
key
Data type: String[1]
The key to retrieve. Must conform to the following:
-
Key must contain only the following characters:
- a-z
- A-Z
- 0-9
- The following special characters:
._:-/
-
Key may not contain '/./' or '/../' sequences.
options
Data type: Optional[Hash]
simpkv configuration that will be merged with
simpkv::options
. All keys are optional.
Options:
-
'app_id'
String
: Specifies an application name that can be used to identify which backend configuration to use via fuzzy name matching, in the absence of thebackend
option.- More flexible option than
backend
. - Useful for grouping together simpkv function calls found in different catalog resources.
- When specified and the
backend
option is absent, the backend will be selected preferring a backend in the mergedbackends
option whose name exactly matches theapp_id
, followed by the longest backend name that matches the beginning of theapp_id
, followed by thedefault
backend. - When absent and the
backend
option is also absent, this function will use thedefault
backend.
- More flexible option than
-
'backend'
String
: Definitive name of the backend to use.- Takes precedence over
app_id
. - When present, must match a key in the
backends
option of the merged options Hash or the function will fail. - When absent in the merged options, this function will select
the backend as described in the
app_id
option.
- Takes precedence over
-
'backends'
Hash
: Hash of backend configurations-
Each backend configuration in the merged options Hash must be a Hash that has the following keys:
type
: Backend type.id
: Unique name for the instance of the backend. (Same backend type can be configured differently).
-
Other keys for configuration specific to the backend may also be present.
-
-
'environment'
String
: Puppet environment to prepend to keys.- When set to a non-empty string, it is prepended to the key used in the backend operation.
- Should only be set to an empty string when the key being accessed is truly global.
- Defaults to the Puppet environment for the node.
-
'softfail'
Boolean
: Whether to ignore simpkv operation failures.- When
true
, this function will return a result even when the operation failed at the backend. - When
false
, this function will fail when the backend operation failed. - Defaults to
false
.
- When
simpkv::list
Type: Ruby 4.x API
Returns a listing of all keys and sub-folders in a folder.
The list operation does not recurse through any sub-folders. Only information about the specified key folder is returned.
Examples
Retrieve the list of key info for a key folder in the default backend
$result = simpkv::list('hosts')
$result['keys'].each |$host, $info | {
host { $host:
ip => $info['value'],
}
}
Retrieve the list of key info for a key folder in the backend servicing an application id
$result = simpkv::list('hosts', { 'app_id' => 'myapp' })
$result['keys'].each |$host, $info | {
host { $host:
ip => $info['value'],
}
}
Retrieve the list of sub-folders in a key folder in the default backend
$result = simpkv::list('applications')
notice("Supported applications: ${join($result['folders'], ' ')}")
Retrieve the top folder list for the environment in the default backend
$result = simpkv::list('/')
Retrieve the list of environments supported by the default backend
$result = simpkv::list('/', { 'environment' => '' })
simpkv::list(String[1] $keydir, Optional[Hash] $options)
Returns a listing of all keys and sub-folders in a folder.
The list operation does not recurse through any sub-folders. Only information about the specified key folder is returned.
Returns: Enum[Hash,Undef]
Hash containing the key/info pairs and list of
sub-folders upon success; Undef when the backend operation fails and
'softfail' is true
in the merged backend options
- 'keys' attribute is a Hash of the key information in the folder
- Each Hash key is a key found in the folder
- Each Hash value is a Hash with a 'value' key and an optional 'metadata' key.
- 'folders' attribute is an Array of sub-folder names
Raises:
ArgumentError
If the key folder or merged backend config is invalidLoadError
If the simpkv adapter cannot be loadedRuntimeError
If the backend operation fails, unless 'softfail' istrue
in the merged backend options.
Examples
Retrieve the list of key info for a key folder in the default backend
$result = simpkv::list('hosts')
$result['keys'].each |$host, $info | {
host { $host:
ip => $info['value'],
}
}
Retrieve the list of key info for a key folder in the backend servicing an application id
$result = simpkv::list('hosts', { 'app_id' => 'myapp' })
$result['keys'].each |$host, $info | {
host { $host:
ip => $info['value'],
}
}
Retrieve the list of sub-folders in a key folder in the default backend
$result = simpkv::list('applications')
notice("Supported applications: ${join($result['folders'], ' ')}")
Retrieve the top folder list for the environment in the default backend
$result = simpkv::list('/')
Retrieve the list of environments supported by the default backend
$result = simpkv::list('/', { 'environment' => '' })
keydir
Data type: String[1]
The key folder to list. Must conform to the following:
-
Key folder must contain only the following characters:
- a-z
- A-Z
- 0-9
- The following special characters:
._:-/
-
Key folder may not contain '/./' or '/../' sequences.
options
Data type: Optional[Hash]
simpkv configuration that will be merged with
simpkv::options
. All keys are optional.
Options:
-
'app_id'
String
: Specifies an application name that can be used to identify which backend configuration to use via fuzzy name matching, in the absence of thebackend
option.- More flexible option than
backend
. - Useful for grouping together simpkv function calls found in different catalog resources.
- When specified and the
backend
option is absent, the backend will be selected preferring a backend in the mergedbackends
option whose name exactly matches theapp_id
, followed by the longest backend name that matches the beginning of theapp_id
, followed by thedefault
backend. - When absent and the
backend
option is also absent, this function will use thedefault
backend.
- More flexible option than
-
'backend'
String
: Definitive name of the backend to use.- Takes precedence over
app_id
. - When present, must match a key in the
backends
option of the merged options Hash or the function will fail. - When absent in the merged options, this function will select
the backend as described in the
app_id
option.
- Takes precedence over
-
'backends'
Hash
: Hash of backend configurations-
Each backend configuration in the merged options Hash must be a Hash that has the following keys:
type
: Backend type.id
: Unique name for the instance of the backend. (Same backend type can be configured differently).
-
Other keys for configuration specific to the backend may also be present.
-
-
'environment'
String
: Puppet environment to prepend to keys.- When set to a non-empty string, it is prepended to the key used in the backend operation.
- Should only be set to an empty string when the key being accessed is truly global.
- Defaults to the Puppet environment for the node.
-
'softfail'
Boolean
: Whether to ignore simpkv operation failures.- When
true
, this function will return a result even when the operation failed at the backend. - When
false
, this function will fail when the backend operation failed. - Defaults to
false
.
- When
simpkv::put
Type: Ruby 4.x API
Sets the data at key
to the specified value
in the configured backend.
Optionally sets metadata along with the value
.
Examples
Set a key using the default backend
simpkv::put("hosts/${facts['clientcert']}", $facts['ipaddress'])
Set a key with metadata using the default backend
$meta = { 'rack_id' => 183 }
simpkv::put("hosts/${facts['clientcert']}", $facts['ipaddress'], $meta)
Set a key with metadata using the backend servicing an application id
$meta = { 'rack_id' => 183 }
$opts = { 'app_id' => 'myapp' }
simpkv::put("hosts/${facts['clientcert']}", $facts['ipaddress'], $meta, $opts)
simpkv::put(String[1] $key, NotUndef $value, Optional[Hash] $metadata, Optional[Hash] $options)
Sets the data at key
to the specified value
in the configured backend.
Optionally sets metadata along with the value
.
Returns: Boolean
true
when backend operation succeeds; false
when the
backend operation fails and 'softfail' is true
in the merged backend
options
Raises:
ArgumentError
If the key or merged backend config is invalidLoadError
If the simpkv adapter cannot be loadedRuntimeError
If the backend operation fails, unless 'softfail' istrue
in the merged backend options.
Examples
Set a key using the default backend
simpkv::put("hosts/${facts['clientcert']}", $facts['ipaddress'])
Set a key with metadata using the default backend
$meta = { 'rack_id' => 183 }
simpkv::put("hosts/${facts['clientcert']}", $facts['ipaddress'], $meta)
Set a key with metadata using the backend servicing an application id
$meta = { 'rack_id' => 183 }
$opts = { 'app_id' => 'myapp' }
simpkv::put("hosts/${facts['clientcert']}", $facts['ipaddress'], $meta, $opts)
key
Data type: String[1]
The key to set. Must conform to the following:
-
Key must contain only the following characters:
- a-z
- A-Z
- 0-9
- The following special characters:
._:-/
-
Key may not contain '/./' or '/../' sequences.
value
Data type: NotUndef
The value of the key
metadata
Data type: Optional[Hash]
Additional information to be persisted
options
Data type: Optional[Hash]
simpkv configuration that will be merged with
simpkv::options
. All keys are optional.
Options:
-
'app_id'
String
: Specifies an application name that can be used to identify which backend configuration to use via fuzzy name matching, in the absence of thebackend
option.- More flexible option than
backend
. - Useful for grouping together simpkv function calls found in different catalog resources.
- When specified and the
backend
option is absent, the backend will be selected preferring a backend in the mergedbackends
option whose name exactly matches theapp_id
, followed by the longest backend name that matches the beginning of theapp_id
, followed by thedefault
backend. - When absent and the
backend
option is also absent, this function will use thedefault
backend.
- More flexible option than
-
'backend'
String
: Definitive name of the backend to use.- Takes precedence over
app_id
. - When present, must match a key in the
backends
option of the merged options Hash or the function will fail. - When absent in the merged options, this function will select
the backend as described in the
app_id
option.
- Takes precedence over
-
'backends'
Hash
: Hash of backend configurations-
Each backend configuration in the merged options Hash must be a Hash that has the following keys:
type
: Backend type.id
: Unique name for the instance of the backend. (Same backend type can be configured differently).
-
Other keys for configuration specific to the backend may also be present.
-
-
'environment'
String
: Puppet environment to prepend to keys.- When set to a non-empty string, it is prepended to the key used in the backend operation.
- Should only be set to an empty string when the key being accessed is truly global.
- Defaults to the Puppet environment for the node.
-
'softfail'
Boolean
: Whether to ignore simpkv operation failures.- When
true
, this function will return a result even when the operation failed at the backend. - When
false
, this function will fail when the backend operation failed. - Defaults to
false
.
- When
simpkv::support::config::merge
Type: Ruby 4.x API
Create merged backend configuration and then validate it.
The merge entails the following operations:
-
The
options
argument is merged withsimpkv::options
Hiera and global simpkv defaults. -
If the
backend
options is missing in the merged options, it is set to a value determined as follows:- If the
app_id
option is present and the name of a backend inbackends
matchesapp_id
,backend
will be set toapp_id
. - Otherwise, if the
app_id
option is present and the name of a backend inbackends
matches the beginning ofapp_id
,backend
will be set to that partially-matching backend name. When multiple backends satisfy the 'start with' match, the backend with the most matching characters is selected. - Otherwise, if the
app_id
option does not match any backend name or is not present,backend
will be set todefault
.
- If the
-
If the
backends
option is missing in the merged options, it is set to a Hash containing a single entry,default
, that has configuration for the simpkv 'file' backend.
Validation includes the following checks:
- configuration for the selected backend exists
- the plugin for the selected backend has been loaded
- different configuration for a specific plugin instance does not exist
simpkv::support::config::merge(Hash $options, Array $backends)
Create merged backend configuration and then validate it.
The merge entails the following operations:
-
The
options
argument is merged withsimpkv::options
Hiera and global simpkv defaults. -
If the
backend
options is missing in the merged options, it is set to a value determined as follows:- If the
app_id
option is present and the name of a backend inbackends
matchesapp_id
,backend
will be set toapp_id
. - Otherwise, if the
app_id
option is present and the name of a backend inbackends
matches the beginning ofapp_id
,backend
will be set to that partially-matching backend name. When multiple backends satisfy the 'start with' match, the backend with the most matching characters is selected. - Otherwise, if the
app_id
option does not match any backend name or is not present,backend
will be set todefault
.
- If the
-
If the
backends
option is missing in the merged options, it is set to a Hash containing a single entry,default
, that has configuration for the simpkv 'file' backend.
Validation includes the following checks:
- configuration for the selected backend exists
- the plugin for the selected backend has been loaded
- different configuration for a specific plugin instance does not exist
Returns: Hash
merged simpkv options that will have the backend to use
specified by 'backend'
Raises:
ArgumentError
if merged configuration fails validation
options
Data type: Hash
Hash that specifies simpkv backend options to be merged with
simpkv::options
.
backends
Data type: Array
List of backends for which plugins have been successfully loaded.
simpkv::support::config::validate
Type: Ruby 4.x API
Validate backend configuration
simpkv::support::config::validate(Hash $options, Array $backends)
Validate backend configuration
Returns: Nil
Raises:
ArgumentError
if a backend has not been specified, appropriate configuration for a specified backend cannot be found, or different backend configurations are provided for the same ['type', 'id'] pair.
options
Data type: Hash
Hash that specifies simpkv backend options
backends
Data type: Array
List of backends for which plugins have been successfully loaded.
simpkv::support::key::validate
Type: Ruby 4.x API
Validates key conforms to the simpkv key specification
-
simpkv key specification
-
Key must contain only the following characters:
- a-z
- A-Z
- 0-9
- The following special characters:
._:-/
-
Key may not contain '/./' or '/../' sequences.
-
-
Terminates catalog compilation if validation fails.
Examples
Passing
simpkv::support::key::validate('looks/like/a/file/path')
simpkv::support::key::validate('looks/like/a/directory/path/')
simpkv::support::key::validate('simp-simp_snmpd:password.auth')
Failing
simpkv::support::key::validate('${special}/chars/not/allowed!'}
simpkv::support::key::validate('looks/like/an/./unexpanded/linux/path')
simpkv::support::key::validate('looks/like/another/../unexpanded/linux/path')
simpkv::support::key::validate(String[1] $key)
Validates key conforms to the simpkv key specification
-
simpkv key specification
-
Key must contain only the following characters:
- a-z
- A-Z
- 0-9
- The following special characters:
._:-/
-
Key may not contain '/./' or '/../' sequences.
-
-
Terminates catalog compilation if validation fails.
Returns: Nil
Raises:
ArgumentError
if validation fails
Examples
Passing
simpkv::support::key::validate('looks/like/a/file/path')
simpkv::support::key::validate('looks/like/a/directory/path/')
simpkv::support::key::validate('simp-simp_snmpd:password.auth')
Failing
simpkv::support::key::validate('${special}/chars/not/allowed!'}
simpkv::support::key::validate('looks/like/an/./unexpanded/linux/path')
simpkv::support::key::validate('looks/like/another/../unexpanded/linux/path')
key
Data type: String[1]
simpkv key
simpkv::support::load
Type: Ruby 4.x API
Load simpkv adapter and plugins and add simpkv 'extension' to the catalog instance, if it is not present
simpkv::support::load()
Load simpkv adapter and plugins and add simpkv 'extension' to the catalog instance, if it is not present
Returns: Nil
Raises:
LoadError
if simpkv adapter software fails to load
- Fri Sep 18 2020 Liz Nemsick lnemsick.simp@gmail.com - 0.7.1
- Advertise EL8 support in metadata.json
- Fixed bad URL to docker's libkv project
- Wed Feb 26 2020 Trevor Vaughan tvaughan@onyxpoint.com - 0.7.0
- Changed name of module from libkv to simpkv
- Tue Oct 01 2019 Liz Nemsick lnemsick.simp@gmail.com - 0.7.0
- Merged in changes released with 0.6.1 and on the 'develop' branch
- Documented libkv requirements (See 'Requirements' section of docs/Design_Protoype2.md).
- Created and documented design for second prototype (See 'Changes from Version 0.6.X' section of docs/Design_Protoype2.md).
- Implemented second prototype and file plugin + store to demonstrate the design
- Wed Jan 31 2018 Dylan Cochran dylan.cochran@onyxpoint.com - 0.6.1
- Release 0.6.1
- Thu Oct 26 2017 Nick Markowski nicholas.markowski@onyxpoint.com - 0.6.0
- (SIMP-3923) Moved libkv::consul to pupmod-simp-simp_consul
- Updated README
- Updated travis with CI credentials
- Wed Oct 25 2017 Dylan Cochran dylan.cochran@onyxpoint.com - 0.5.0
- (SIMP-3445) Add acceptance test for consul configuration and bootstrap
- (SIMP-3629) libkv::atomic_put returns false for a successful put on consul >0.9.0
- Thu Aug 24 2017 Dylan Cochran dylan.cochran@onyxpoint.com - 0.4.3
- (SIMP-3623) libkv::lookup_key backend should turn softfail on by default
- Add fix for 0.9.x consul installations
- Tue Jul 18 2017 Dylan Cochran dylan.cochran@onyxpoint.com - 0.4.2
- (SIMP-3001) Prevent '.' and '..' from being used in keys
- (SIMP-3446) Add parameters to reconfigure http and https listen
- (SIMP-3429) libkv::list isn't always removing keys
- Tue Jul 18 2017 Dylan Cochran dylan.cochran@onyxpoint.com - 0.4.1
- Always copy over consul-acl, and update metadata
- Tue Jul 18 2017 Dylan Cochran dylan.cochran@onyxpoint.com - 0.4.0
- (SIMP-3275) libkv auto-config uses the root acl
- Tue Jul 11 2017 Dylan Cochran dylan.cochran@onyxpoint.com - 0.3.3
- (SIMP-3406) Fix docker containers for travisci
- (SIMP-3128) Delete .meta keys
- Tue Jul 11 2017 Dylan Cochran dylan.cochran@onyxpoint.com - 0.3.2
- (SIMP-3407) Fix idempoency on acl token generation
- (SIMP-3403) Spurious 'undefined method unpack'
- Mon Jul 10 2017 Dylan Cochran dylan.cochran@onyxpoint.com - 0.3.1
- (SIMP-3360) Use module data for certificate paths
- (SIMP-3087) Add libkv::lookup hierav5 backend function
- Mon Jul 10 2017 Dylan Cochran dylan.cochran@onyxpoint.com - 0.3.0
- (SIMP-2961) Add automatic cluster creation for consul.
- (SIMP-3130) metadata needs to default to 'String'.
- (SIMP-3129) atomic_create needs to create metadata
- (SIMP-3127) libkv can't list / since metadata update
- (SIMP-3122) Move the libkv wrapper outside the loader
- (SIMP-3125) Move key regex match into libkv wrapper
- (SIMP-3110) Use .meta to convert a value to the correct type
- (SIMP-3060) Fix travisci tests
- (SIMP-3109) Create a .meta key to store type
- Sat Apr 29 2017 Dylan Cochran dylan.cochran@onyxpoint.com - 0.2.0
- (SIMP-2978) Fix readme generation
- (SIMP-3019) Add ssl/tls support to consul backend
- (SIMP-2962) Add consul_members fact
- (SIMP-3097) Add 'serialize' and 'mode' parameters to all libkv functions
- (SIMP-3102) Flesh out spec tests
- (SIMP-2964) Add generic 'libkv::auth' parameter
- Fri Jan 6 2017 Dylan Cochran dylan.cochran@onyxpoint.com - 0.1.0
- Initial release
Dependencies
- puppetlabs/stdlib (>= 4.9.0 < 7.0.0)
simpkv - Per Section 105 of the Copyright Act of 1976, these works are not entitled to domestic copyright protection under US Federal law. The US Government retains the right to pursue copyright protections outside of the United States. The United States Government has unlimited rights in this software and all derivatives thereof, pursuant to the contracts under which it was developed and the License under which it falls. --- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.