puppet-st2

Module to manage StackStorm

Currently tested with:

• Ubuntu 14.04
• RHEL/Centos 6/7

Compatability currently being tested:

• Debian 7
• Ubuntu 12.04

Maintainers

• James Fryman james@stackstorm.com
• Patrick Hoolboom patrick@stackstorm.com

Quick Start

For a full installation on a single node, a profile already exists to get you setup and going with minimal effort. Simply:

include ::st2::profile::fullinstall

Ubuntu 14.04

Because 14.04 ships with a very old version of puppet (3.4) and most puppet modules no longer support this version of puppet, we recommend upgrading to 3.8.7 at a minimum.

# 14.04 trusty
# By default this ships with puppet 3.4.x (very old), need a newer version to 
# work with with of the required puppet module dependencies for this module. 
wget https://apt.puppetlabs.com/puppetlabs-release-trusty.deb
sudo dpkg -i puppetlabs-release-trusty.deb
sudo apt-get update
sudo apt-get install puppet=3.8.7-1puppetlabs1

Configuration

This module aims to provide sane default configurations, but also stay out of your way in the event you need something more custom. To accomplish this, this module uses the Roles/Profiles pattern. Included in this module are several modules that come with sane defaults that you can use directly or use to compose your own site-specific profile for StackStorm installation.

Configuration can be done directly via code composition, or set via Hiera data bindings. A few notable parameters to take note of:

• st2::version - Version of ST2 to install. (Latest version w/o value)

All other classes are documented with Puppetdoc. Please refer to specific classes for use and configuration.

Profiles:

• st2::profile::client - Profile to install all client libraries for st2
• st2::profile::fullinstall - Full installation of StackStorm and dependencies
• st2::profile::mistral - Install of OpenStack Mistral
• st2::profile::mongodb - st2 configured MongoDB installation
• st2::profile::nodejs - st2 configured NodeJS installation
• st2::profile::python - Python installed and configured for st2
• st2::profile::rabbitmq - st2 configured RabbitMQ installation
• st2::proflle::server - st2 server components
• st2::profile::web - st2 web components

Installing and configuring Packs

StackStorm packs can be installed and configured directly from Puppet. This can be done via the st2::pack and st2::pack::config defined types.

Installation/Configuration via modules:

  # install pack from the exchange
  st2::pack { 'linux': }
  
  # install pack from a git URL
  st2::pack { 'private':
    repo_url => 'https://private.domain.tld/git/stackstorm-private.git',
  }
  
  # install pack and apply configuration
  st2::pack { 'slack':
    config   => {
      'post_message_action' => {
        'webhook_url' => 'XXX',
      },
    },
  }

Installation/Configuration via Hiera:

st2::packs:
  linux:
    ensure: present
  private:
    ensure: present
    repo_url: https://private.domain.tld/git/stackstorm-private.git
  slack:
    ensure: present
    config:
      post_message_action:
        webhook_url: XXX

Known Limitations

MongoDB (all OSes)

When running the initial install of st2 you will see an error from the MongoDB module :

Error: Could not prefetch mongodb_database provider 'mongodb': Could not evaluate MongoDB shell command: load('/root/.mongorc.js'); printjson(db.getMongo().getDBs())

This error is caused by a deficiency in this module trying to use authentication in its prefetch step when authentication hasn't been configured yet on the database. The error can be safely ignored. Auth and databases will be configured normally. Subsequent runs of puppet will not show this error.

Ubuntu 16.04

Due to a known bug in st2 3290 when first running the installation with this puppet module the st2 pack will fail to install. Simply invoking puppet a second time will produce a fully running st2 installation with the st2 pack installed.

Module Dependencies

RHEL 7 Notes (Ruby 2.0.0, Puppet 3.8.7)

The following modules need to have their versions locked in your Puppetfile because future versions dropped support for Puppet 3.x. All other dependencies are compatible with Puppet 3 (as of 2017-08-03).

mod 'puppet/nginx', '0.6.0'
mod 'puppetlabs/vcsrepo', '1.5.0'
mod 'puppet/nodejs', '2.3.0'

RHEL 6 Notes (Ruby 1.8.7, Puppet 3.8.6)

The following modules need to have their versions locked in your Puppetfile because future versions dropped support for Puppet 3.x. All other dependencies are compatible with Puppet 3 (as of 2017-08-03).

mod 'puppet/nginx', '0.6.0'
mod 'puppetlabs/vcsrepo', '1.5.0'
mod 'puppet/nodejs', '1.3.0'

Note that puppet/nodejs is an older version than for RHEL 7. This is because 1.3.0 dropped compatibility with Ruby 1.8.7 in future versions. If you run a version >1.3.0 on with Ruby 1.8.7, then you'll encounter the following error.

Error: Could not autoload puppet/provider/package/npm: /var/lib/puppet/lib/puppet/provider/package/npm.rb:3: syntax error, unexpected ':', expecting $end
...package).provide :npm, parent: Puppet::Provider::Package do
                              ^
Error: Could not retrieve local facts: Could not autoload puppet/provider/package/npm: /var/lib/puppet/lib/puppet/provider/package/npm.rb:3: syntax error, unexpected ':', expecting $end
...package).provide :npm, parent: Puppet::Provider::Package do
                              ^
Error: Failed to apply catalog: Could not retrieve local facts: Could not autoload puppet/provider/package/npm: /var/lib/puppet/lib/puppet/provider/package/npm.rb:3: syntax error, unexpected ':', expecting $end
...package).provide :npm, parent: Puppet::Provider::Package do
                              ^

Ubuntu Notes (Puppet 3.8.7)

The following modules need to have their versions locked in your Puppetfile because future versions dropped support for Puppet 3.x. All other dependencies are compatible with Puppet 3 (as of 2017-08-03).

mod 'puppet/nginx', '0.6.0'
mod 'puppetlabs/vcsrepo', '1.5.0'
mod 'puppet/nodejs', '2.3.0'
mod 'puppetlabs/apt', '2.4.0'

Dev Notes

Ubuntu dev environment

sudo apt-get install ruby-dev git gcc g++ make
gem install bundler

# 14.04 trusty
# By default this ships with puppet 3.4.x (very old), need a newer version to work
# with any of the required puppet modules (minimum version = 3.8.x)
wget https://apt.puppetlabs.com/puppetlabs-release-trusty.deb
sudo dpkg -i puppetlabs-release-trusty.deb
sudo apt-get update
sudo apt-get install puppet=3.8.7-1puppetlabs1

# 16.04 xenial
# Note: because the version of ruby shipped with Xenail is 2.3.x and the version
# of puppet shipped (3.8.x) is incompatible, we have to run our tests using
# a newer version of puppet (4.10 at a minimum) that supports the new version of
# ruby.
# Note - this is ONLY a unit testing deficiency, the usage of this module runs
# just fine with the default version of puppet.
wget https://apt.puppetlabs.com/puppetlabs-release-pc1-xenial.deb
sudo dpkg -i puppetlabs-release-pc1-xenial.deb
sudo apt-get update
sudo apt-get install puppet-agent

RHEL dev environment

sudo yum -y install puppet ruby-devel git gcc g++ make
gem install bundler

Dev testing (standalone box)

The file docs/Puppetfile is an r10k Puppetfile that downloads all of the dependencies required for this module and allows you to perform standalone testing without a puppet master. To utilize this file and form of testing perform the following

# assuming you already have your dev environment setup from above (puppet must be installed)

# install r10k
gem install puppet_forge:2.2.6 r10k

# run r10k to download all of our module dependencies defined in ./docs/Puppetfile
r10k puppetfile install -v --moduledir=./modules --puppetfile=./docs/Puppetfile

# run StackStorm full install using puppet
puppet apply --modulepath=./modules -e "include ::st2::profile::fullinstall"

How to generate Gemfile.lock.x.y.x

TODO Install and switch to the proper versions of ruby before each call (chruby)

gem install bundler
PUPPET_VERSION=3.8.7 bundle package; mv Gemfile.lock Gemfile.lock.3.8.7
PUPPET_VERSION=4.10 bundle package; mv Gemfile.lock Gemfile.lock.4.10
PUPPET_VERSION=5.0 bundle package; mv Gemfile.lock Gemfile.lock.5.0

Changelog

1.0.0-beta (Aug 14, 2017)

files/repo/nodesource/NODESOURCE-GPG-SIGNING-KEY-EL

• Removed unused file after cleaning up nodejs profiel (Enhancement)

manifests/auth/standalone.pp

• Did not have access to the ::st2 variables (Bugfix).
• Had a dependency issue where (on some platforms) allowed the htpasswd file to be created after the st2 services were starting (Bugfix)
• Created an unnecessary "test user" (Bugfix)

manifests/auth_user.pp

• Dependency issues here where the htpasswd file was sometimes trying to be created before the /etc/st2 directory was created, and other times it was trying to be created after the st2 services had started. (Bugfix)

manifests/init.pp

• Needed extra variables for SSL setup in st2web. (Feature)
• Needed extra variables for proper database setup (mongodb and postgres) (Enhancement)
• Needed path to the st2auth logging config file (Enhancement)
• Needed variables about the datastore encryption keys (Feature)

manifests/kv.pp

• Some puppet lint problems (notice the whitespace fix and reordering of class params) (Bugfix)
• Dependency issues where the tag being used for the Service resource was incorrect (Bugfix)
• Dependency issues where sometimes st2 hadn't been reloaded so the k/v loads would fail (Bugfix)

manifests/notices.pp

• Puppet lint fixes for using double quotes without variable interpolation in the string. (Bugfix)

manifests/pack.pp

• Unit tests revealed that many of the dependencies of this resource were not declared (group and directories) (Bugfix)
• Pointing at old location for config directory (Bugfix)
• Needed lots of dependency work to ensure resources were created in the proper order (Bugfix)

manifests/params.pp

• Broke down the old st2_server_packages variable into various components to align more with what ansible-st2 and the "one liner" shell scripts do in their functions. (Enhancement)
• Removed some unused code in the "init provider" section (Enhancement)
• Broke down the old st2_services into its components similar to st2_server_packages. FYI: The mistral services are handled by the mistral install instead of being grouped together into st2 server. (Enhancement)
• Added lots of new parameters for services that were not configured in the past like (nginx, st2web, mongodb, rabbitmq) (Feature)

manifests/profile/client.pp

• Removed stale comment (Enhancement)

manifests/profile/fullinstall.pp

• Mainly dependency cleanup here. (Bugfix)
• Ensure that packages are installed in the correct order and that there are meaningful anchors in place in case others need to execute tasks at certain points during the install. (Bugfix)

manifests/profile/mistral.pp

• This was completely re-written (Enhancement)
• Previously it was performing a lot of tasks manually that i believe st2mistral package now handles for us (Enhancement)

manifests/profile/mongodb.pp

• Completely re-written (Enhancement)
• It now handles auth (did not previously) (Enhancement)
• It also deals with several deficiencies in the puppetlabs-mongodb module. This module has lots of annoying bugs. I'm not at the point where i want to code up a new module myself yet, but we do have to work around several quirks for this to even work (sorry!). (Bugfix)

manifests/profile/nginx.pp

• New profile that installs and configures nginx (does not setup st2web config, that is left to the st2web profile) (Feature)
• Utilizes the nginx puppet module to do all of the heavy lifting here (Feature)

manifests/profile/nodejs.pp

• Completely re-written (Enhancement)
• Utilizes the nodejs puppet module to do all of the heavy lifting instead of doing it ourselves (Enhancement)
• Works around a small quirk of the module on RedHat distributions (BugFix)

manifests/profile/postgresql.pp

• Expanded this to properly configure postgres for listening according to the standard installs (shell scripts and ansible-st2) (Enhancement)
• Also ensured that 9.4 is installed on RHEL6 (Bugfix)

manifests/profile/rabbitmq.pp

• Greatly simplified by allowing the rabbitmq module to do all of the heavy lifting for us (Enhancement)

manifests/profile/repos.pp

• Fixed a bug where we were pointing to an all lowercase URL which caused st2 package installs to fail (Bugfix)

manifests/profile/selinux.pp

• Added a class that configures SELinux on RHEL hosts (Feature)

manifests/profile/server.pp

• Small changes here related to adding database auth capability (Enhancement)
• Added stanley user creation (Feature)
• Added datastore crypto creation (Feature)
• Added additional dependency management (Bugfix)

manifests/profile/web.pp

• Completely re-written (Enhancement)
• I don't believe that st2web was complete when this module was last touched, so this class got a complete overhaul (Enhancement)

manifests/rbac.pp

• Fixed a few puppet lint errors (Bugfix)
• Fixed an error where the RBAC rules were executed every puppet run (Bugfix)

manifests/server/datastore_keys.pp

• New manifest that manages the datastore crypto keys (Feature)

manifests/stanley.pp

• Removed unnecessary warning about ssh keys (Bugfix)

manifests/user.pp

• Fixed a couple small bugs related to a legacy "robots" group. (Bugfix)
• This got a pretty big overhaul with regards to SSH key creation. Now, if SSH keys are not present new ones will be created (just like the shell scripts and ansible-st2) (Bugfix)

metadata.json

• Reformatted the whole file to standard JSON formatting scheme (Enhancement)
• Updated module dependencies (some were missing) (Bugfix)
• Added supported OS block (Enhancement)
• Added supported puppet versions block (Enhancement)

spec/*

• Lots of small fixes here related to running the tests on various versions of ruby. (Bugfix)
• Finally found a happy medium where all tests now pass (Bugfix)
• Removed tests for the "st2::package::debian" type that no longer exists (Bugfix)

templates/*

• Removed the following unused templates due to code cleanup and modernizaiton (Enhancement)
  • templates/etc/init.d/mistral-api.erb
  • templates/etc/init.d/mistral.erb
  • templates/etc/init/mistral-api.conf.erb
  • templates/etc/init/mistral.conf.erb
  • templates/etc/init/st2actionrunner-worker.conf.erb
  • templates/etc/systemd/system/mistral-api.service.erb
  • templates/etc/systemd/system/mistral.service.erb
  • templates/etc/systemd/system/st2actionrunner.service.erb
  • templates/etc/systemd/system/st2service_multi.service.erb
  • templates/etc/systemd/system/st2service_single.service.erb
  • templates/opt/st2web/config.js.erb

0.14.1 (Jan 15, 2015)

• Fix typo - st2garbagecollector is part of st2reactor package.

0.14.0 (Jan 15, 2015)

• Add services files for the new st2garbagecollector service.

0.13.0 (Jan 8, 2015)

• Don't install a default SSH key for stanley user if one is not explicitly provided.

0.12.3 (Dec 15, 2015)

• Adding tests around bintray repo feature

0.12.2 (Dec 11, 2015)

• Fixing error where WebUI fails because of missing resource

0.12.1 (Dec 9, 2015)

• Adding ability to download packages from testing bintray repos

0.11.1 (Dec 9, 2015)

• Extract a new version of st2web on update

0.11.0 (Dec 4, 2015)

• Force rewrite of webui/config.js on every provision

0.10.18 (Nov 11, 2015)

• Disable upstart logging for st2 services.
• Make sure that st2web logs on Ubuntu under upstart are written to /var/log/st2web.log

0.10.17 (Nov 2, 2015)

• Parameterized download server to CI

0.10.16 (Oct 30, 2015)

• Set sticky bit on Group, not User for stackstorm packs dir

0.10.15 (Oct 28, 2015)

• Remove DAG errors with fullinstall profile

0.10.14 (Oct 26, 2015)

• Ensure /opt/stackstorm/packs directory is SGID for pack group

0.10.13 (Oct 22, 2015)

• Add st2packs to default deploy and ensure Stanley exists
• Limit setting of api_url to st2::helper::auth_manager

0.10.8 (Oct 21, 2015)

• Adding api_url parameter to server profile

0.10.7 (Oct 21, 2015)

• Adding backend kwargs attribute to st2::helper::auth_manager
• Disable static UID for auto-generated users

0.10.4 (Oct 19. 2015)

• Fix for RHEL 6 client package installation
• Re-enable ng_init env flag to compat with st2ctl
• Fix issue with actionrunners outputting to STDOUT/STDERR
• All SysV init scripts ensure sourcing from /etc/environment

0.10.1 (Oct 16. 2015)

• Init scripts default install now

0.10.0 (Oct 15, 2015)

• Bug fixes
• Service restart with Ini_setting { tag => 'st2::config' }

0.9.19 (Oct 14, 2015)

• Repair init scripts for Mistral on RHEL 6/7 and Debian

0.9.17 (Oct 14, 2015)

• Repair package map with CentOS 7 systems

0.9.15 (Oct 13, 2015)

• Support for SysV and SystemD Init types

0.9.14 (Oct 2, 2015)

• Ensure postgresql is setup and running before starting Mistral service.

0.9.12 (Oct 1, 2015)

• Refresh services on ini setting change.

0.9.11 (Oct 1, 2015)

• Add ability for user to change SSH key location in /etc/st2/st2.conf

0.9.10 (Sept 28, 2015)

• Fix typo in RBAC type.

0.9.8 (Sept 25, 2015)

• Add ability to manage StackStorm RBAC roles (improvement)

0.9.7 (Sept 22, 2015)

• Restart mistral on init script update

0.9.6 (Sept 22, 2015)

• Add silence_ssl_warnings option to the client profile.

0.9.5 (Sept 21, 2015)

• pin stahnma-epel to 1.1.0

0.9.4 (Sept 18, 2015)

• Restart services on package update (bugfix)

0.9.3 (Sept 17, 2015)

• Fix condition where autoupdate: false would result in missing resources (bugfix)

0.9.2 (Sept 17, 2015)

• Configure WebUI to integrate with Flow (feature)
• Configure st2client CLI settings for any user (improvement)

0.9.0 (Sept 16, 2015)

• Add support for RHEL/CentOS 6 & 7

0.8.0 (Sept 10, 2015)

• Release StackStorm v0.13.2
• Stop st2::pack resource restarting StackStorm (improvement)

0.7.10 (Sept 2, 2015)

• Fix manage_mysql -> manage_postgresql in st2::profile::server (bugfix)
• Fix error with stanley user UID change (bugfix)

0.7.9 (Sept 1, 2015)

• Fix path for logging config with st2auth subsystem (bugfix)

0.7.8 (Aug 30, 2015)

• Allow user to adjust username of 'st2::stanley' resource (improvement)

0.7.7 (Aug 29, 2015)

• Bump default StackStorm version to 0.13.1 (upgrade)