Forge Home

Modules

Resources

PuppetEcosystemPuppet CommunityUpdates

clickhouse

Installs, configures and manages Clickhouse server and client
Project URLRSS FeedReport issues

1,506 downloads

119 latest version

5.0 quality score

We run a couple of automated
scans to help you access a
module's quality. Each module is
given a score based on how well
the author has formatted their
code and documentation and
modules are also checked for
malware using VirusTotal.

Please note, the information below
is for guidance only and neither of
these methods should be considered
an endorsement by Puppet.

Version information

  • 2.0.0 (latest)
  • 1.2.0
  • 1.1.0
released Feb 14th 2024
This version is compatible with:
  • Puppet Enterprise 2023.6.x, 2023.5.x, 2023.4.x, 2023.3.x, 2023.2.x, 2023.1.x, 2023.0.x, 2021.7.x, 2021.6.x, 2021.5.x, 2021.4.x, 2021.3.x, 2021.2.x, 2021.1.x, 2021.0.x, 2019.8.x, 2019.7.x, 2019.5.x, 2019.4.x, 2019.3.x, 2019.2.x, 2019.1.x, 2019.0.x, 2018.1.x, 2017.3.x, 2017.2.x, 2016.4.x
  • Puppet >= 4.10.0 < 9.0.0
  • , , ,

Start using this module

  • r10k or Code Manager
  • Bolt
  • Manual installation
  • Direct download

Add this module to your Puppetfile:

mod 'deric-clickhouse', '2.0.0'
Learn more about managing modules with a Puppetfile

Add this module to your Bolt project:

bolt module add deric-clickhouse
Learn more about using this module with an existing project

Manually install this module globally with Puppet module tool:

puppet module install deric-clickhouse --version 2.0.0

Direct download is not typically how you would use a Puppet module to manage your infrastructure, but you may want to download the module in order to inspect the code.

Download

Documentation

deric/clickhouse — version 2.0.0 Feb 14th 2024

Puppet Clickhouse

Puppet Forge Tests

The clickhouse module installs, configures and manages the Clickhouse Server service.

It also allows to create and manage users, quotas, profiles, dictionaries, databases as well as configure replication and sharding.

Setup

Setup Requirements

This module requires xml-simple gem, which is used to translate Hash configuration to Clickhouse XML format configuration files. To install it you need to execute following command on your puppetmaster server:

sudo puppetserver gem install xml-simple

Beginning with clickhouse

To install a server with the default options:

include 'clickhouse::server'.

To customize Clickhouse server options, you must also pass in an override hash:

class { 'clickhouse::server':
  override_options => $override_options
}

See Customize Server Options below for examples of the hash structure for $override_options.

Usage

All server configuration is done via clickhouse::server. To install client separatly, use clickhouse::client

Customize server options

To define server options, pass a hash structure of overrides in clickhouse::server. Server configuration parameters can be found at https://clickhouse.yandex/docs/en/operations/server_settings/settings/

$override_options = {
  parent_xml_tag => {
    item => thing,
  }
}

For example, to configure zstd compression pass following override options hash:

$override_options = {
  compression => {
    case => {
      min_part_size       => 10000000000,
      min_part_size_ratio => 0.01,
      method              => 'zstd',
    }
  }
}

It will add following section to Clickhouse configuration file:

<compression>
    <case>
        <min_part_size>10000000000</min_part_size>
        <min_part_size_ratio>0.01</min_part_size_ratio>
        <method>zstd</method>
    </case>
</compression>

Create a database

To create a database:

clickhouse_database { 'my_database':
  ensure => 'present',
}

Specify passwords for users

In addition to passing passwords in plain text, you can input them in sha256 encoding. For example:

class { 'clickhouse::server':
  users => {
    myuser => {
      password => '02472d6a1e23f73b37481bbd67949a5d16cbaf3d71770696f20a0cd773a2e682',
    }
  }
}

Install Clickhouse Server

This example shows how to install Clickhouse Server, configure zstd compression for it, create production profile and quota, create user alice and assign production profile and quota to her and also configure replication and three different sharding schemas: replicated - just a basic replication with single shard and two replicas; segmented - two shards without replicas; segmented_replicated - two shards each having two replicas:

class { 'clickhouse::server':
  override_options => {
    compression => {
      case => {
        min_part_size       => 10000000000,
        min_part_size_ratio => 0.01,
        method              => 'zstd',
      },
    },
  },
  profiles         => {
    production => {
      use_uncompressed_cache => 0,
      log_queries => 1,
      max_memory_usage => ceiling($facts['memory']['system']['total_bytes'] * 0.7),
    },
  },
  quotas           => {
    production => {
      interval => [
        {
          duration       => 3600,
          queries        => 200,
          erros          => 10,
          result_rows    => 0,
          read_rows      => 0,
          execution_time => 0,
        }
      ],
    },
  },
  users            => {
    alice => {
      password => '02472d6a1e23f73b37481bbd67949a5d16cbaf3d71770696f20a0cd773a2e682',
      quota    => 'production',
      profile  => 'production',
      network  => {
        ip => ['::/0'],
      },
    },
  },
  replication      => {
    zookeeper_servers => ['zookeeper1.local:2181', 'zookeeper2.local:2181', 'zookeeper3.local:2181'],
    macros => {
      cluster => 'Clickhouse_cluster',
      replica => $facts['networking']['fqdn'],
    },
  },
  remote_servers   => {
    replicated           => {
      shard => {
        internal_replication => true,
        replica              => ['host1.local:9000', 'host2.local:9000'],
      },
    },
    segmented            => {
      shard1 => {
        weight               => 1,
        internal_replication => true,
        replica              => ['host1.local:9000'],
      },
      shard2 => {
        weight               => 2,
        internal_replication => true,
        replica              => ['host2.local:9000'],
      },
    },
    segmented_replicated => {
      shard1 => {
        internal_replication => true,
        replica              => ['host1.local:9000', 'host2.local:9000'],
      },
      shard2 => {
        internal_replication => true,
        replica              => ['host3.local:9000', 'host4.local:9000'],
      },
    },
  },
}

Hiera support

Configuration from YAML backend:

clickhouse::server::override_options:
  listen_host: 0.0.0.0
  default_profile: default
  default_database: default
  mark_cache_size: 5368709120
  uncompressed_cache_size: 8589934592
  path: /var/lib/clickhouse/
  interserver_http_port: 9009
  interserver_http_host: "%{::fqdn}"
  max_connections: 4096
  max_concurrent_queries: 100
  http_port: 8123
  tcp_port: 9000
  prometheus:
    endpoint: /metrics
    port: 8001
    metrics: true
    events: true
    asynchronous_metrics: true
clickhouse::server::profiles:
  default:
    max_memory_usage: 10000000000 #  Maximum memory usage for processing single query, in bytes
    use_uncompressed_cache: 0 # Use cache of uncompressed blocks of data. Meaningful only for processing many of very short queries
    load_balancing: random
  readonly:
    readonly: 1

Replicas

Configure host properties:

clickhouse::server::remote_servers:
  replicated:
    shard:
      internal_replication: true
      replicas:
        host1.local:
          port: 9000
          priority: 1
          default_database: foo
          tcp_port_secure: 9100
          secure: 1
          user: click
          password: house
        host2.local:
          port: 9000

Crash reports

Settings for opt-in sending crash reports.

clickhouse::server::crash_reports:
  enabled: true
  endpoint: http://sentry.localhost # Overrides the Sentry endpoint
  debug: false

Reference

Classes

Public Classes

Private Classes

  • clickhouse::client::install: Private class for managing Clickhouse client package.
  • clickhouse::params: Private class for setting default Clickhouse parameters.
  • clickhouse::server::config: Private class for Clickhouse server configuration.
  • clickhouse::server::install: Private class for managing Clickhouse server package.
  • clickhouse::server::resources: Private class for applying Clickhouse resources.
  • clickhouse::server::service: Private class for managing the Clickhouse service.

Defined types

Resource types

Functions

Limitations

For a list of supported operating systems, see metadata.json

Development

Please feel free to fork, modify, create issues, bug reports and pull requests.

Testing epp templates

Render template from CLI:

$ puppet epp render templates/zookeeper.xml.epp --values "{'zookeeper_servers' => ['172.0.0.1:2181'], distributed_ddl=> {path => '/foo' }}"