clickhouse
Version information
This version is compatible with:
- Puppet Enterprise 2019.8.x, 2019.7.x, 2019.5.x, 2019.4.x, 2019.3.x, 2019.2.x, 2019.1.x, 2019.0.x, 2018.1.x, 2017.3.x
- Puppet >= 5.0 <= 7.0.0
- , , , , ,
Start using this module
Add this module to your Puppetfile:
mod 'innogames-clickhouse', '1.0.1'Learn more about managing modules with a PuppetfileDocumentation
Description
This module provides an easy way to install and configure ClickHouse DBMS. The easiest way to do the magic is:
class { 'clickhouse':
server => true,
manage_repo => true,
}
Setup
The module depends on xml-simple gem. It allows to use clickhouse::server::config and clickhouse::client::config to manage configuration with puppet Hash. To install it execute the next command on your puppet server:
sudo puppetserver gem install xml-simple
Usage
See the examples
Reference
Classes
clickhouse: this class allows you to install ClickHouse DB's repo, client and serverclickhouse::client: ClickHouse client classclickhouse::repo: installs repository with ClickHouse DBMSclickhouse::server: ClickHouse server classclickhouse::server::config::memory: Configure memory consumptionclickhouse::server::config::zookeeper: Set proper zookeeper config
Defined types
clickhouse::client::config: generates xml config from hash via ruby xml-simpleclickhouse::error: Implements error logging with continue of manifests applicationclickhouse::server::config: generates xml config from hash via ruby xml-simpleclickhouse::server::config::profile: Define ClickHouse profileclickhouse::server::config::user: Define ClickHouse users
Resource types
clickhouse_database: Manages databases on ClickHouse server
Limitations
ClickHouse does work only on UNIX-like OS. Theoretically possible to use this module on FreeBSD and Mac OS but should be checked additionally. Feel free to test and open an issue/PR.
The current state of module was tested with Centos 7, Ubuntu 18.04 and Debian 9.
Development
Fork this project, develop, make pull request and wait for corresponding auto tests will be done.
Reference
Table of Contents
Classes
clickhouse: this class allows you to install ClickHouse DB's repo, client and serverclickhouse::client: ClickHouse client classclickhouse::repo: installs repository with ClickHouse DBMSclickhouse::server: ClickHouse server classclickhouse::server::config::default_localhost: Restricts network access fordefaultclickhouse::server::config::memory: Configure memory consumptionclickhouse::server::config::zookeeper: Set proper zookeeper config
Defined types
clickhouse::client::config: generates xml config from hash via ruby xml-simpleclickhouse::error: Implements error logging with continue of manifests applicationclickhouse::server::config: generates xml config from hash via ruby xml-simpleclickhouse::server::config::profile: Define ClickHouse profileclickhouse::server::config::user: Define ClickHouse users
Resource types
clickhouse_database: Manages databases on ClickHouse server
Classes
clickhouse
Top level class for ClickHouse DBMS installation and management.
Examples
Simple use
include clickhouse
Install server and client
class { 'clickhouse':
server => true,
}
Install everything and manage repository
class { 'clickhouse':
server => true,
client => true,
manage_repo => true,
}
Parameters
The following parameters are available in the clickhouse class.
server
Data type: Boolean
Is clickhouse-server should be installed or not. It won't be installed by default, because some time you need the client only.
Default value: false
client
Data type: Boolean
Is clickhouse-client should be installed or not.
Default value: true
manage_repo
Data type: Boolean
Is apt or yum repository should be managed. Set to false by default to not affect your own repositories policy.
Default value: false
user
Data type: String[1]
User for configs owning. Strongly recommended staying default.
It's highly unrecommended to change default user, but feel free to shot in a knee. You have to adjust as well the data directory and the whole configs everywhere.
Default value: 'clickhouse'
group
Data type: String[1]
Group for configs, see user parameter.
Default value: $user
clickhouse::client
The class installs clickhouse-client.
Examples
Simple use
include clickhouse::client
Use with params
class { 'clickhouse::client':
package_name => 'clickhouse-client-custom',
package_ensure => 'hold',
user => 'custom-user',
config_d_dir => '/some/path',
}
Parameters
The following parameters are available in the clickhouse::client class.
package_name
Data type: String[1]
Package to be installed.
Default value: 'clickhouse-client'
package_ensure
Data type: String[1]
Client package ensure. See ensure attribute for package resource.
Default value: 'installed'
conf_d_dir
Data type: Optional[Stdlib::Unixpath]
Deprecated, use config_d_dir.
Default value: undef
config_d_dir
Data type: Stdlib::Unixpath
Directory for custom configs. Unmanaged configs will be removed from the dirrectory during puppet running.
Default value: $conf_d_dir
clickhouse::repo
Installs yum or apt repository for ClickHouse DBMS.
Uses Altinity repository for yum and yandex for apt.
Examples
Simple use
include clickhouse::repo
clickhouse::server
The class installs ClickHouse server and manages service
Examples
Simple use
include clickhouse::server
Use with params
class { 'clickhouse::server':
package_name => 'clickhouse-server-custom',
package_ensure => 'latest',
service_name => 'clickhouse-server',
service_ensure => false,
service_enable => false,
}
Parameters
The following parameters are available in the clickhouse::server class.
package_name
Data type: String[1]
Server package to be installed.
Default value: 'clickhouse-server'
package_ensure
Data type: String[1]
Server package ensure. See ensure attribute for package resource.
Default value: 'installed'
service_name
Data type: String[1]
Name of the managed service for clickhouse-server.
Default value: $package_name
service_ensure
Data type: Variant[Boolean, Enum[ 'running', 'stopped' ]]
Desired state for $service_name, see ensure for service resource.
Default value: 'running'
service_enable
Data type: Variant[Boolean, Enum[ 'manual', 'mask' ]]
If $service_name should be enabled, see enable for service resource.
Default value: true
config_service_notify
Data type: Boolean
If true, every config managed by this module and requires for server restart will trigger service refresh.
Default value: true
conf_dir
Data type: Optional[Stdlib::Unixpath]
Deprecated, use config_dir
Default value: undef
config_dir
Data type: Stdlib::Unixpath
Directory with clickhouse-server configuration.
Default value: $conf_dir
conf_d_dir
Data type: Optional[Stdlib::Unixpath]
Deprecated, use config_d_dir
Default value: undef
config_d_dir
Data type: Stdlib::Unixpath
Directory with clickhouse-server included configuration. Unmanaged configs will be removed from the directory during puppet run.
Default value: $conf_d_dir
users_d_dir
Data type: Stdlib::Unixpath
Directory with clickhouse-server users configuration. Unmanaged configs will be removed from the directory during puppet run.
Default value: "${config_dir}/users.d"
clickhouse::server::config::default_localhost
clickhouse::server::config::default_localhost
By default, ClickHouse listens only localhost and user default is able to connect from anywhere.
This class, if included, restricts network access.
Examples
Simple use
include clickhouse::server::config::default_localhost
clickhouse::server::config::memory
clickhouse::server::config::memory This class provide basic memory adjustment for keep ClickHouse server in reasonable limits
- See also
Examples
Simple use
include clickhouse::server::config::memory
Parameters
The following parameters are available in the clickhouse::server::config::memory class.
reserved_memory
Data type: Integer[0]
Memory to leave for system usage and CH overheads like merges and mutations
Default value: 4294967296
mark_cache_size
Data type: Integer[0]
See CH documentation
Default value: 5368709120
uncompressed_cache_size
Data type: Optional[Integer[0]]
See CH documentation
Default value: undef
max_memory_usage_for_all_queries
Data type: Optional[Integer[0]]
See CH documentation
Default value: undef
max_memory_usage
Data type: Optional[Integer[0]]
See CH documentation
Default value: undef
use_uncompressed_cache
Data type: Boolean
See CH documentation
Default value: true
external_group_by
Data type: Boolean
See CH documentation
Default value: true
external_sort
Data type: Boolean
See CH documentation
Default value: true
memory_check
Data type: Boolean
Check if class has proper parameters and raise errors otherwise
Default value: true
service_notify
Data type: Boolean
If ClickHouse server should be restarted on the config update
Default value: $clickhouse::server::config_service_notify
clickhouse::server::config::zookeeper
clickhouse::server::config::zookeeper Set proper zookeeper config
Examples
Use with params
class { 'clickhouse::server::config::zookeeper':
nodes => { 'server1' => 1},
user => 'user',
password => 'password',
}
Parameters
The following parameters are available in the clickhouse::server::config::zookeeper class.
nodes
Data type: Hash[String[1], Integer[1, 255]]
hash of zk_server: id to define nodes
port
Data type: Integer[1, 65536]
cilent port of zookeeper cluster
Default value: 2181
session_timeout_ms
Data type: Integer[1]
maximum timeout for client session in milliseconds
Default value: 30000
operation_timeout_ms
Data type: Integer[1]
maximum timeout for operation in milliseconds
Default value: 10000
root
Data type: Optional[Stdlib::Unixpath]
ZNode, that is used as root for znodes used by ClickHouse server
Default value: undef
user
Data type: Optional[String[1]]
user if zookeeper uses authorization
Default value: undef
password
Data type: Optional[String[1]]
password if zookeeper uses authorization
Default value: undef
mode
Data type: String[1]
zookeeper nodes, ports and optional user and password are confidential data and normally should not be readable
Default value: '0440'
service_notify
Data type: Boolean
If ClickHouse server should be restarted on the config update
Default value: $clickhouse::server::config_service_notify
Defined types
clickhouse::client::config
This type creates custom configuration files for clickhouse-client.
Examples
Usage
clickhouse::client::config { 'prompt':
data => {'prompt_by_server_display_name' => [{
'experimental' => ['{display_name} \x01\e[1;35m\x02:)\x01\e[0m\x02 '],
}]},
}
#
# Will create file `/etc/clickhouse-client/conf.d/prompt.xml`:
# <config>
# <prompt_by_server_display_name>
# <experimental>{display_name} \x01\e[1;35m\x02:)\x01\e[0m\x02 </experimental>
# </prompt_by_server_display_name>
# </config>
Parameters
The following parameters are available in the clickhouse::client::config defined type.
data
Data type: Hash
This hash will be converted into xml config placed in $clickhouse::client::config_d_dir.
Root will be <config> by default.
ensure
Data type: Enum[ 'present', 'file', 'absent' ]
Subset of attribute ensure for file resource.
Default value: 'present'
mode
Data type: String[1]
Desired permissions mode for the config file, see mode attribute for file resource.
Default value: '0644'
clickhouse::error
clickhouse::error
Implements error which cause 6 exit code with --detailed-exitcodes argument for puppet run.
Work around notifying about severe problem
Examples
Simple use
clickhouse::error { 'Error message':
}
clickhouse::server::config
This type creates custom configuration files for clickhouse-server.
Examples
Usage
clickhouse::server::config { 'macros':
data => {'macros' => [
{'hostname' => ['host.domain.TLD'],}
]},
section => 'config',
}
#
# Will create file `/etc/clickhouse-server/conf.d/macros.xml`:
# <yandex>
# <macros>
# <hostname>host.domain.TLD</hostname>
# </macros>
# </yandex>
Parameters
The following parameters are available in the clickhouse::server::config defined type.
data
Data type: Hash
This hash will be converted into xml config placed in $clickhouse::server::config_d_dir.
Root will be <yandex> by default.
section
Data type: Enum['config', 'users']
Should the file be created in configs or users include directory.
ensure
Data type: Enum[ 'present', 'file', 'absent' ]
Subset of attribute ensure for file type.
Default value: 'present'
mode
Data type: String[1]
Desired permissions mode for the config file, see mode attribute for file resource.
Default value: '0644'
service_notify
Data type: Boolean
If service should be restarted on the config changing.
Default value: false
clickhouse::server::config::profile
This type defines the ClickHouse profile with specified options
Examples
clickhouse::server::config::profile { 'profile_name':
databases => {
'db_name' => {
'table_name' => ['filter'],
'another_table_name' => ['another filter'],
},
},
}
Parameters
The following parameters are available in the clickhouse::server::config::profile defined type.
profile
Data type: String[1]
User profile
Default value: $title
settings
Data type: Hash
Profile settings
clickhouse::server::config::user
This type defines the ClickHouse users with specified parameters
- See also
Examples
clickhouse::server::config::user { 'username':
network => {
'ip' => [
'::',
'0.0.0.0',
],
'host' => [
'host1.local',
'host2.local',
],
'host_regexp' => [
'[^.]*\.domain\.TLD',
],
},
profile => 'profile_name',
quota => 'quota_name',
password => 'password',
databases => {
'db_name' => {
'table_name' => ['filter'],
'another_table_name' => ['another filter'],
},
},
}
Parameters
The following parameters are available in the clickhouse::server::config::user defined type.
networks
Data type: Struct[{ Optional[ip] => Array[String[1], 1], Optional[host] => Array[String[1], 1], Optional[host_regexp] => Array[String[1], 1], }]
Defines network nodes where from user could connect
profile
Data type: String[1]
User profile
Default value: 'default'
quota
Data type: String[1]
User quota
Default value: 'default'
user
Data type: String[1]
User name
Default value: $title
password
Data type: String[0]
Plaintext password. Will be hashed into sha256 format in xml file
Default value: ''
password_sha256
Data type: Optional[ Pattern[/\A[0-9a-fA-F]{64}\Z/] ]
Optional parameter. If defined, $password must not be defined
Default value: undef
databases
Data type: Optional[Hash]
Optional hash of allowed DBs, tables and filters to implement ACLs
Default value: undef
allow_databases
Data type: Optional[Array[String[1]]]
Optional parameter to restrict access to specified databases
Default value: undef
allow_dictionaries
Data type: Optional[Array[String[1]]]
Optional parameter to restrict access to specified dictionaries
Default value: undef
Resource types
clickhouse_database
Manages databases on ClickHouse server
Properties
The following properties are available in the clickhouse_database type.
engine
Valid values: Lazy, MySQL, Ordinary
Engine of the database
Default value: Ordinary
ensure
Valid values: present, absent
CREATE or DROP DATABASE
Default value: present
Parameters
The following parameters are available in the clickhouse_database type.
engine_settings
This parameter depends on the engine parameter:
Lazy: must be Integer[1]
MySQL: must be Array[String, 4, 4]
Ordinary: must be undefined
See: https://clickhouse.tech/docs/en/engines/database_engines/
force
Valid values: true, false, yes, no
DROP database even if it contains tables
If ENGINE=MySQL for an existing database, it will be dropped anyway since it's just a connector
Default value: false
name
namevar
Name of the database
provider
The specific backend to use for this clickhouse_database resource. You will seldom need to specify this --- Puppet
will usually discover the appropriate provider for your platform.
Dependencies
- innogames-xmlsimple (>=1.0 <2.0)
- puppetlabs-stdlib (>= 4.20.0 <5.2.0)
MIT License Copyright (c) 2018 InnoGames GmbH Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.