fts

Table of Contents

1. Description
2. Setup - The basics of getting started with fts
3. Usage - Configuration options
4. Limitations - OS compatibility

Description

A module to install and configure an FTS3 server and its MySQL database on a couple of Hosts.

Setup

• Setup two machines with CentOS 7 (puppet agents, not constrained to CentOS 7);
• Install puppet agent on both machines:
• CentOS:

1. add the puppet repo rpm -Uvh https://yum.puppetlabs.com/puppet7/puppet7-release-el-7.noarch.rpm;

2. install the puppet agent and vim packages (yum install -y puppet-agent vim)

• Ubuntu:

1. wget https://apt.puppet.com/puppet7-release-focal.deb
2. sudo dpkg -i puppet7-release-focal.deb
3. sudo apt update && sudo apt install puppet-agent vim

• Reload your /etc/profile to update $PATH or run puppet agent by absolute path
• create the certificates folder in the root path and copy there your site hostcert.pem and hostkey.pem;
• Employ the fts class to configure the FTS server and the MySQL database on the nodes, see the fts=db.pp and fts-server.pp in examples for some reference;
• Add the following modules in your puppetfile:

mod 'cnafsd-voms', '0.8.0'
mod "puppetlabs-stdlib",'9.4.0'
mod 'puppet-cron', '4.1.0'
mod 'puppet-selinux', '4.0.0'
mod 'puppetlabs-inifile', '6.1.0'
mod 'puppet-systemd', '6.0.0'
mod 'puppetlabs-firewall', '7.0.2'
mod 'puppet/yum', '7.1.0'
mod 'puppetlabs/apache', '11.1.0'
mod 'puppetlabs/mysql', '15.0.0'
mod 'bradipoeremita-concat', '9.0.0'
mod 'nasd-fts', '0.1.6'

• Modify the puppet.conf (vim /etc/puppetlabs/puppet/puppet.conf) file in both servers to point to your puppet server with the fts_development environment

[main]
        server = yourpuppetserver.infn.it

[agent]
        pluginsync  = true
        report      = true
        environment = fts_development

• execute puppet agent -t -v on both servers (db first, fts server second);
• execute fetch-crl (it is possible to use -p parameter to parallelize download);
• restart httpd: systemctl restart httpd

Test your installation

1. Check your monitoring webpage at https://{fts_fqdn}:8449/fts3/ftsmon/#/;
2. Execute the following command on your server curl --capath /etc/grid-security/certificates -E /etc/grid-security/hostcert.pem --key /etc/grid-security/hostkey.pem https://fts_host:8446/whoami;

Usage

The module can be used to configure both the database and fts server on the same machine, or on two different machines. For the latter use case, to configure the fts server only, set configure_db to false, and to configure the mysql database only, set configure_fts to false. The exaples folder contains several exemplary manifests.

Limitations

It works only on CentOS 7 distributions.

Reference

Table of Contents

Classes

• fts: This class installs the FTS3 server and configures it to run as a service. It can also install the MySQL server and create the FTS3 database. The class can be used to configure only the FTS3 server, only the MySQL server, or both.
• fts::client: This class isntall the FTS client
• fts::database: @summary: this class can create and configures the mysql fts database. Depending on the parameter choices, it can create an mysql server, c
• fts::server: This class defines the configuration for the FTS server. It takes in parameters for configuring the server and sets up the necessary resources.

Classes

fts

(optional) The number of threads to use for the FTS3 database. defaults to 24.

Examples

class { 'fts':
  fts_host           => 'fts3-server.example.org',
  db_host            => 'fts3-db.example.org',
  db_name            => 'fts',
  db_root_user       => 'root',
  db_root_password   => 'roottestpassword',
  fts_db_password    => 'ftstestpassword',
  fts_db_user        => 'fts3',
  fts_db_type        => 'mysql',
  fts_server_alias   => 'fts3-server',
  admin_list         => ['/DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Michele Delli Veneri delliven@infn.it'],
  fts_db_threads_num => 24,
  configure_db       => true,
  configure_fts      => true,
  configure_firewall => true,
  configure_selinux  => true,
  build_mysql_server => true,
  build_fts_tables   => true,
  grant_privileges   => true,
  configure_admins   => true,
  configure_lsc      => true,
  vo_list            => ['alice', 'atlas', 'cms', 'cygno', 'datacloud', 'dteam', 'escape', 'lhcb', 'ops', 'wlcg'],
}

Parameters

The following parameters are available in the fts class:

• fts_host
• db_host
• db_name
• db_root_user
• db_root_password
• fts_db_password
• fts_db_type
• fts_server_alias
• admin_list
• fts_db_threads_num
• configure_fts
• configure_db
• configure_firewall
• configure_admins
• configure_lsc
• configure_selinux
• build_mysql_server
• vo_list
• build_fts_tables
• grant_privileges
• fts_db_user

fts_host

Data type: String

(required) The hostname of the FTS3 server. defaults to fts-server.infn.it The value of this parameter is used to set the FTS3 server hostname in the MySQL database.

Default value: 'fts3-server.infn.it'

db_host

Data type: String

(required) The hostname of the FTS3 database. Defaults to fts-db.infn.it The value of this parameter is used to set the FTS database hostname in the FTS configuration file.

Default value: 'fts3-db.infn.it'

db_name

Data type: String

(optional) the name of the fts database user. defaults to fts3. The user will be created if it does not exist.

Default value: 'fts'

db_root_user

Data type: String

(optional) The username of the MySQL root user. defaults to 'root'. If the mysql server is not built, or grants to the root and fts users must not be given becouse the database alredy exists, this parameter is ignored.

Default value: 'root'

db_root_password

Data type: String

(optional) the root password for the mysql server. Defaults to roottestpassword. If the mysql server is not built, or grants to the root and fts users must not be given becouse the database alredy exists, this parameter is ignored.

Default value: 'dbrootpassword'

fts_db_password

Data type: String

(optional) the password of the fts database user. defaults to ftstestpassword. Please change this parameter to a secure password.

Default value: 'ftstestpassword'

fts_db_type

Data type: String

(optional) The type of database backend to use. defaults to mysql which is the only supported backend.

Default value: 'mysql'

fts_server_alias

Data type: String

(optional) The alias to use for the FTS server defaults to fts3-server.

Default value: 'fts3-server'

admin_list

Data type: Array

(required) the list of the admin users for the fts database. defaults to ['/DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Michele Delli Veneri] In order for the fts server to work, at least one admin user must be configured. The admin user must be in the form of a DN. Admins will be created if they do not exist only if the FTS database has been populated with tables through the build_fts_tables parameter.

Default value: ['/DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Michele Delli Veneri delliven@infn.it']

fts_db_threads_num

Data type: Integer

Default value: 24

configure_fts

Data type: Boolean

(optional) Whether to configure the FTS3 server. defaults to true. If this parameter is set to True, the FTS3 server will be configured, i.e. the class fts::server will be included and used to install needed dependencies and configure the FTS3 service. For further information about the parameters of the fts::server class, please refer to the documentation of the class.

Default value: true

configure_db

Data type: Boolean

(optional) Whether to configure the FTS3 MySQL database. defaults to true. If this parameter is set to True, the FTS3 database will be configured, i.e. the class fts::database will be included and used to install needed dependencies and configure the MySQL FTS3 database. For further information about the parameters of the fts::database class, please refer to the documentation of the class.

Default value: true

configure_firewall

Data type: Boolean

(optional) Whether to configure the firewall. defaults to true. If configure_fts is set to true, the firewallthe firewall module opens the following ports: 8446 for the REST API, 8449 for the web monitoring. If configure_db is set to true, the firewall module opens the following ports: 3306 for the MySQL server. All remaining ports are closed.

Default value: true

configure_admins

Data type: Boolean

(required) Whether to configure the FTS3 administrators. defaults to true. If set to true, the list of admins is taken from the admin_list parameter, and the admins are created if they do not exist.

Default value: true

configure_lsc

Data type: Boolean

(optional) Whether to install and configure the servers as VOMS clients. defaults to true. If set to false, VOMS must be already configured on the server. LSC setup for at least one VO (and one admin for that VO) is required for the FTS3 server to work.

Default value: true

configure_selinux

Data type: Boolean

(optional) Whether to configure SELinux. defaults to true. Selinux is set to enforcing on the FTS server, and to permissive on the MySQL server.

Default value: true

build_mysql_server

Data type: Boolean

(optional) whether to build the mysql server or not. defaults to true. if the mysql server is not built, the script assumes that a mysql server is already running on the machine and that the root user and password are valid.

Default value: true

vo_list

Data type: Array

(optional) List of VOs to configure. Add the VOs to the list. Possible values are 'alice', 'atlas', 'cms', 'cygno', 'datacloud', 'dteam', 'escape', 'lhcb', 'ops', 'wlcg' defaults to [None].

Default value: ['cycgno', 'datacloud']

build_fts_tables

Data type: Boolean

(optional) Whether to build the FTS tables or not. defaults to true. The script in either case will create and/or check the presente of the fts database and the user. If the parameter is set to true, the fts database will be populated with the tables needed for the fts server to work. If the parameter is set to false, the script will only check the presence of the fts database and the user. The tables can be build both from the fts server through an automatic SQL query on the MySQL database (in case the MySQL is CentOS 7), or directly on the mySQL server by combining this flag with configure_db = true.

Default value: true

grant_privileges

Data type: Boolean

(optional) Whether to grant privileges to the FTS and root user on the database. defaults to true. In order to grant privileges, the MySQL database, the FTS Tables, and user must already exist and the MySQL root user and password must be provided. Correct privileges to the fts database for, at least, the fts user are neeed for the fts server to work. So, if the parameter is set to false, make sure to grant privilegs manually.

Default value: true

fts_db_user

Data type: String

Default value: 'fts3'

fts::client

A description of what this class does

Examples

include fts::client

fts::database

@summary: this class can create and configures the mysql fts database. Depending on the parameter choices, it can create an mysql server, create and configure the fts database and the user, populate it with tables, add admins, and configure the firewall and selinux.

Examples

Configure the fts database

class { 'fts::database':
  db_root => 'root',
  db_root_password => 'ftstestpassword',
  db_name     => 'fts',
  fts_host    => 'fts-server.infn.it',
  fts_db_user => 'fts3',
  fts_db_password => 'ftstestpassword',
  admin_list  => ['/DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Michele Delli Veneri,
                    '/DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Massimo Sgaravatto sgaravat@infn.it'],
  configure_firewall => true,
  configure_selinux  => true,
  build_mysql_server => true,
  build_fts_tables   => true,
  grant_privileges   => true,
}

Parameters

The following parameters are available in the fts::database class:

• db_root_user
• db_root_password
• db_name
• fts_host
• fts_db_user
• fts_db_password
• admin_list
• configure_firewall
• configure_selinux
• build_mysql_server
• build_fts_tables
• grant_privileges
• configure_admins

db_root_user

Data type: String

(optional) the root user for the mysql server, defaults to root. If the mysql server is not built, or grants to the root and fts users must not be given becouse the database alredy exists, this parameter is ignored.

Default value: 'root'

db_root_password

Data type: String

(optional) the root password for the mysql server. Defaults to roottestpassword. If the mysql server is not built, or grants to the root and fts users must not be given becouse the database alredy exists, this parameter is ignored.

Default value: 'roottestpassword'

db_name

Data type: String

(required) the name of the fts database. defaults to fts. The database will be created if it does not exist.

Default value: 'fts'

fts_host

Data type: String

(required) the hostname of the fts server. This can be the FQDN or the IP address of the machine hosting the mysql db.

Default value: 'fts-server.infn.it'

fts_db_user

Data type: String

(required) The user that will run the FTS server. defaults to fts3.

Default value: 'fts3'

fts_db_password

Data type: String

(optional) the password of the fts database user. defaults to ftstestpassword. Please change this parameter to a secure password.

Default value: 'ftstestpassword'

admin_list

Data type: Array

(required) the list of the admin users for the fts database. In order for the fts server to work, at least one admin user must be configured. The admin user must be in the form of a DN. Admins will be created if they do not exist only if the FTS database has been populated with tables through the build_fts_tables parameter.

Default value: ['/DC=org/DC=terena/DC=tcs/C=IT/O=Istituto Nazionale di Fisica Nucleare/CN=Michele Delli Veneri delliven@infn.it']

configure_firewall

Data type: Boolean

(optional) whether to configure the firewall or not. defaults to true. The firewall will be configured to allow access only to the mysql server.

Default value: true

configure_selinux

Data type: Boolean

(optional) whether to configure selinux or not. defaults to true. Selinux will be configured to permissive mode.

Default value: true

build_mysql_server

Data type: Boolean

(optional) whether to build the mysql server or not. defaults to true. if the mysql server is not built, the script assumes that a mysql server is already running on the machine and that the root user and password are valid.

Default value: true

build_fts_tables

Data type: Boolean

(optional) Whether to build the FTS tables or not. d defaults to true. The script in either case will create and/or check the presente of the fts database and the user. If the parameter is set to true, the fts database will be populated with the tables needed for the fts server to work. If the parameter is set to false, the script will only check the presence of the fts database and the user.

Default value: true

grant_privileges

Data type: Boolean

(optional) Whether to grant privileges to the FTS and root user on the database. defaults to true. In order to grant privileges, the MySQL database, the FTS Tables, and user must already exist and the MySQL root user and password must be provided. Correct privileges to the fts database for, at least, the fts user are neeed for the fts server to work. So, if the parameter is set to false, make sure to grant privilegs manually.

Default value: true

configure_admins

Data type: Boolean

(optional) Whether to configure the FTS admins or not. defaults to true. In order to configure the admins, the MySQL database, the FTS Tables, and user must already exist.

Default value: true

fts::server

}

o n a CentOS machine, otherwise building the tables must be done manually or by running t he module on the database machine.

Examples

class { 'fts::server':
  fts_user => 'fts3',
  fts_db_type => 'mysql',
  db_host => 'fts-db.infn.it',
  fts_db_name => 'fts',
  fts_db_password => 'ftstestpassword',
  fts_db_threads_num => 24,
  fts_server_alias => 'fts3-server',
  configure_firewall => true,

Parameters

The following parameters are available in the fts::server class:

• fts_user
• fts_db_type
• db_host
• fts_db_name
• fts_db_password
• fts_db_threads_num
• fts_server_alias
• configure_firewall
• configure_selinux
• build_fts_tables

fts_user

Data type: String

(required) The user that will run the FTS server. defaults to fts3.

Default value: 'fts3'

fts_db_type

Data type: String

(optional) The type of database backend to use. defaults to mysql which is the only supported backend.

Default value: 'mysql'

db_host

Data type: String

(required) The hostname or IP of the machine hosting the mysql database.. defaults to fts-db.infn.it. This host must be accessible from the FTS server.

Default value: 'fts-db.infn.it'

fts_db_name

Data type: String

(optional) The name of the mysql database hosted on the database server. defaults to fts.

Default value: 'fts'

fts_db_password

Data type: String

(optional) The password for the fts user to connect to the database. defaults to ftstestpassword.

Default value: 'ftstestpassword'

fts_db_threads_num

Data type: Integer

(optional) The number of threads to use for the database backend. defaults to 24.

Default value: 24

fts_server_alias

Data type: String

(optional) The alias to use for the FTS server defaults to fts3-server.

Default value: 'fts3-server'

configure_firewall

Data type: Boolean

(optional) Whether to configure the firewall or not. defaults to true. If set to false, the firewall must be configured manually. If set to true, the firewall module opens the following ports: 8446 for the REST API, 8449 for the web monitoring.

Default value: true

configure_selinux

Data type: Boolean

(optional) Whether to configure SELinux or not. defaults to true. If set to true, it sets SELinux to enforcing mode.

Default value: true

build_fts_tables

Data type: Boolean

(optional) Whether to build the FTS tables or not. defaults to true. In order to build the tables, the MySQL fts database, and user must already exist. It can only be done if the mysql server is hosted

Default value: true

Changelog

All notable changes to this project will be documented in this file.

Release 0.1.0

Features

Bugfixes

Known Issues