cflogsink

Description

Centralized logging infrastructure.

What it does:

• Use ElasticSearch+Logstash stack for centralized log sink.
  • Advanced tuning for all cf* modules.
  • Dedicated strictly types system log, HTTP access log and firewall log indexes.
  • Special handling of various log format cases.
  • Optimized storage field.
  • Special handling of known message formats to minimize noise.
  • Automatic error detection in regular log level.
• Reliable real-time message delivery through rsyslog RELP.
  • Proxy host support for complex configurations
• Supports UDP & TCP local receive (suitable for JVM services).
• Heavy duty /dev/hdlog skipping systemd.
  • Mostly for HTTP access log and similar.
  • UDP & TCP are also available.
• NetFilter LOG (NFLOG/ULOG) support through ulogd2
• CLI ElastciSearch log viewer cflog:
  • Help for cases when Kibana gets broken or not available
  • Mimics ordinary log file output
  • Supports simple queries for filtering
• Kibana installation is provided in cfwebapp module.

Technical Support

• Example configuration
• Free & Commercial support: support@codingfuture.net

Setup

Up to date installation instructions are available in Puppet Forge: https://forge.puppet.com/codingfuture/cflogsink

Please use librarian-puppet or cfpuppetserver module to deal with dependencies.

There is a known r10k issue RK-3 which prevents automatic dependencies of dependencies installation.

Examples

Please check codingufuture/puppet-test for example of a complete infrastructure configuration and Vagrant provisioning.

Implicitly created resources

cfnetwork::describe_services:
    # if $cflogsink::repo_proxy
    'aptproxy':
        server: "tcp/${proxy_port}"
    # if cflogsink::netsyslog
    netsyslog:
        server:
            - 'tcp/514'
            - 'udp/514'
    # if cflogsink::hdsyslog
    hdsyslog:
        server:
            - 'tcp/${port}'
            - 'udp/${port}'
cfnetwork::service_ports:
    # if cflogsink::netsyslog
    'local:netsyslog': {}
    # if cflogsink::hdsyslog
    'local:hdsyslog': {}
cfnetwork::client_ports:

Class parameters

cflogsink class

The main Hiera-friendly configuration class

• $iface = $cfsystem::service_face - interface for server instance to listen.
• $server = false - cflogsink::endpoint option hash or just true to enable main logsink endpoint.
• $target = false - cflogsink::client option hash or just string with hostname of centralized log sink.
• $tls = undef - controls if connection to target has to be secured via TLS.

cflogsink::netsyslog class

Adds local UDP & TCP syslog sockets for regular use of JVM and other services which do not support UNIX domain socket.

cflogsink::hdsyslog class

Heavy Duty syslog provided through /dev/hdlog. Sutiable for HTTP access logs and similar load. This functionality skips systemd and is designed to work in pair with cflogsink::client.

• $tune = {} - allow rsyslog ruleset queue tuning
• $port = 515 - port to use for UDP & TCP

cflogsink::client class

Centralized logging client setup.

• $host = ... - auto-configured based on cflogsink::target.
• $port = ... - auto-configured based on cflogsink::target main` endpoint.
• $tls = ... - auto-configured based on explicit cflogsink::tls. Otherwise, checks if location mismatch between client and target.
• $timeout = 90 - session timeout.
• $conn_timeout = 5 - connection timeout.
• tls_compress = .. - auto-configured.
• $tune = {} - fine tune rsyslog:
  • queue.* - go to main_queue()
  • rest goes into action() config

cflogsink::endpoint type

Configure log sink endpoint.

• $type = 'logstash' - type of endpoint, only logstash is supported so far.
• $config = undef - override the default configuration template.
• $memory_weight = 100 - memory weight for automatic distribution.
• $memory_max = undef - max memory to use in MB.
• $cpu_weight = 100 - CPU weight for scheduling.
• $io_weight = 100 - IO weight for scheduling.
• $settings_tune = {} - fine tune generated configuration.
• $iface = $cflogsink::iface - interface to bind.
• $port = undef - port to use for insecure connections.
• $secure_port = undef - port to use for TLS connections.
• $internal_port = undef - port to use for internal connections.
• $dbaccess = undef - database acccess (e.g. Elasticsearch for Logstash).
• $extra_clients = [] - extra list of insecure clients (for cfnetwork::ipset).
• $extra_secure_clients = [] - extra list of secure clients (for cfnetwork::ipset).

cflog_${title} CLI tool

The tool is created per cflogsink::endpoint instance. Below is example for the default one. All output goes to less which scrolls to end by default.

It's essential in case of emergency when Kibana output is not available.

By default size limit is 10000 messages. They are counted from the newest. Amount of skipped messages can be seet with <from> argument.

Day, month and year selection can be done through index name.

Usage:

Usage: cflog_main <index> [<query> [<from> [<size>]]]
Known indexes: 'access', 'fw' and 'log'

Lookup latest logs for all hosts:

$ cflog_main log
...
2018-03-15T16:04:05.824Z        web.example.com notice  Received disconnect from 10.0.2.2 port 39233:11: disconnected by user
2018-03-15T16:04:05.824Z        web.example.com notice  Disconnected from 10.0.2.2 port 39233
2018-03-15T16:04:05.825Z        web.example.com notice  pam_unix(sshd:session): session closed for user vagrant
....
2018-03-15T16:04:13.802Z        web2.example.com        notice  Disconnected from 10.0.2.2 port 53173
2018-03-15T16:04:13.803Z        web2.example.com        notice  pam_unix(sshd:session): session closed for user vagrant
...
2018-03-15T16:04:56.815Z        puppet.example.com      notice  rexec line 25: Deprecated option RhostsRSAAuthentication
...

Lookup firewall logs for particular month and host:

$ cflog_main fw-2018.03 host:maint.example.com
... 
2018-03-15T16:10:11.397Z        maint.example.com       OUT-unknown: IN= OUT=eth1 MAC= SRC=:: DST=ff02::16 LEN=96 TC=0 HOPLIMIT=1 FLOWLBL=0 PROTO=ICMPv6 TYPE=143 CODE=0
2018-03-15T16:10:11.397Z        maint.example.com       OUT-unknown: IN= OUT=eth1 MAC= SRC=:: DST=ff02::16 LEN=96 TC=0 HOPLIMIT=1 FLOWLBL=0 PROTO=ICMPv6 TYPE=143 CODE=0
2018-03-15T16:10:11.397Z        maint.example.com       OUT-unknown: IN= OUT=eth1 MAC= SRC=:: DST=ff02::16 LEN=76 TC=0 HOPLIMIT=1 FLOWLBL=0 PROTO=ICMPv6 TYPE=143 CODE=0
2018-03-15T16:10:11.397Z        maint.example.com       OUT-unknown: IN= OUT=eth1 MAC= SRC=:: DST=ff02::16 LEN=76 TC=0 HOPLIMIT=1 FLOWLBL=0 PROTO=ICMPv6 TYPE=143 CODE=0
2018-03-15T16:10:11.397Z        maint.example.com       OUT-vagrant: IN= OUT=eth0 MAC= SRC=:: DST=ff02::16 LEN=76 TC=0 HOPLIMIT=1 FLOWLBL=0 PROTO=ICMPv6 TYPE=143 CODE=0
2018-03-15T16:10:11.397Z        maint.example.com       OUT-vagrant: IN= OUT=eth0 MAC= SRC=:: DST=ff02::16 LEN=76 TC=0 HOPLIMIT=1 FLOWLBL=0 PROTO=ICMPv6 TYPE=143 CODE=0
2018-03-15T16:10:13.125Z        maint.example.com       OUT-vagrant: IN= OUT=eth0 MAC= SRC=fe80::a00:27ff:fe8d:c04d DST=ff02::16 LEN=76 TC=0 HOPLIMIT=1 FLOWLBL=0 PROTO=
2018-03-15T16:10:13.125Z        maint.example.com       OUT-main: IN= OUT=eth1 MAC= SRC=fe80::a00:27ff:fea8:e56a DST=ff02::16 LEN=96 TC=0 HOPLIMIT=1 FLOWLBL=0 PROTO=ICM
2018-03-15T16:10:13.125Z        maint.example.com       OUT-vagrant: IN= OUT=eth0 MAC= SRC=fe80::a00:27ff:fe8d:c04d DST=ff02::16 LEN=76 TC=0 HOPLIMIT=1 FLOWLBL=0 PROTO=
2018-03-15T16:10:13.125Z        maint.example.com       OUT-main: IN= OUT=eth1 MAC= SRC=fe80::a00:27ff:fea8:e56a DST=ff02::16 LEN=96 TC=0 HOPLIMIT=1 FLOWLBL=0 PROTO=ICM

Lookup access logs for particular application:

$ cflog_main access app:cfpuppetserver
...
2018-03-15T16:07:44.803Z                200     130     puppetback.example.com  GET /puppet/v3/node/db.example.com?environment=production&configured_environment=product
2018-03-15T16:07:44.941Z                200     36      puppetback.example.com  GET /puppet/v3/file_metadatas/pluginfacts?environment=production&links=follow&recurse=tr
2018-03-15T16:07:45.885Z                200     729     puppetback.example.com  GET /puppet/v3/file_metadatas/plugins?environment=production&links=follow&recurse=true&s
2018-03-15T16:07:46.175Z                200     24      puppetback.example.com  GET /puppet/v3/file_content/plugins/puppet/provider/cflogsink_endpoint/cflogsink.rb?envi
2018-03-15T16:07:46.230Z                200     24      puppetback.example.com  GET /puppet/v3/file_content/plugins/puppet/provider/cflogsink_endpoint/cflogsink.rb?envi
2018-03-15T16:07:46.357Z                200     27      puppetback.example.com  GET /puppet/v3/file_metadatas/locales?environment=production&links=follow&recurse=true&s
2018-03-15T16:08:10.808Z                200     23096   puppetback.example.com  POST /puppet/v3/catalog/db.example.com?environment=production
2018-03-15T16:08:29.039Z                200     2467    puppetback.example.com  PUT /puppet/v3/report/db.example.com?environment=production&

1.3.2 (2019-11-13)

• FIXED: Debian librelp0 priority

1.3.1 (2019-06-17)

• CHANGED: returned mmutf8fix before Logstash
• FIXED: cflog_* query to handle spaces
• FIXED: workaround for rsyslog 8.1905 segfault
• NEW: 'proxy' mode for cflogsink::endpoint
• NEW: MongoDB message triage support

1.3.0 (2019-04-14)

• CHANGED: to terminate RELP in rsyslog for all cases
• CHANGED: got rid of historical cf-apt-update workaround
• FIXED: cflog_{} access to properly show clientip
• FIXED: to use the latest rsyslog for imrelp endpoint (MSGID corruption fix)

1.1.0 (2018-12-09)

• CHANGED: updated for Ubuntu 18.04 Bionic support

1.0.1 (2018-04-29)

• FIXED: missing module hiera.yaml
• FIXED: multiple logstash instance DB access name conflict
• FIXED: manifest error when logstash is bound to 'local' iface
• FIXED: rsyslog.conf parsing issue logstash endpoint and secure clients

0.12.3 (2018-03-24)

• CHANGED: strip of app name & most kv.* fields in firewall logs
• FIXED: to show app name in cflog_* tool
• FIXED: minor Puppet warnings
• FIXED: missing geoip.location field
• FIXED: to use mmutf8fix to workaround some artifacts in imrelp+TLS
• NEW: in/out/blacklist/forward tagging of firewall logs
• NEW: custom elasticsearch templates for each log type

0.12.2 (2018-03-19)

• CHANGED: to use cf_notify for warnings

0.12.1 (2018-03-15)

Initial release