Forge Home

ct2ls

Installs a daemon script to import logs from cloudtrail to logstash

10,011 downloads

9,030 latest version

4.5 quality score

Version information

  • 0.1.0 (latest)
  • 0.0.2
  • 0.0.1
released Mar 23rd 2015
This version is compatible with:
  • , ,

Start using this module

  • r10k or Code Manager
  • Bolt
  • Manual installation
  • Direct download

Add this module to your Puppetfile:

mod 'evenup-ct2ls', '0.1.0'
Learn more about managing modules with a Puppetfile

Add this module to your Bolt project:

bolt module add evenup-ct2ls
Learn more about using this module with an existing project

Manually install this module globally with Puppet module tool:

puppet module install evenup-ct2ls --version 0.1.0

Direct download is not typically how you would use a Puppet module to manage your infrastructure, but you may want to download the module in order to inspect the code.

Download

Documentation

evenup/ct2ls — version 0.1.0 Mar 23rd 2015

Why do I want this?

You run some or all of your environment in Amazon Web Services, want to enable CloudTrail to maintain audit logs of what is going on, and want those logs to (almost) magically appear in logstash.

How does it work?

Amazon has added the ability to log all API calls via the CloudTrail service. When CloudTrail is enabled, it writes log files to a S3 bucket with the option of adding a notification via SNS when new logs are dropped. Adding a SQS receiver to the SNS topic allows you to poll for new log files that have been dropped.

A simple script included with this puppet module subscribes to the SQS queue, grabs the referenced log file from S3, formats the log messages for logstash and adds them to a redis list for consumption by logstash.

A sample AWS policy would look like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1400869009000",
      "Effect": "Allow",
      "Action": [
        "sqs:GetQueueUrl",
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage"
      ],
      "Resource": [
        "arn:aws:sqs:<region>:<account_ID>:<queue_name>"
      ]
    }, {
      "Sid": "Stmt1400852117000",
      "Effect": "Allow",
      "Action": [
        "s3:DeleteObject",
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::<bucket_name>/*"
      ]
    }
  ]
}

Usage

This script is conveniently wrapped in a puppet module to simplify installation.

Known Issues:

Only tested on CentOS 6

TODO:


[ ] Make evenup/ruby module optional

License:


Released under the Apache 2.0 licence

Contribute:

  • Fork it
  • Create a topic branch
  • Improve it with your awesome ideas or fix my dumb bugs
  • Push new topic branch
  • Submit a PR