AWS Kinesis Firehose Setup for Data Reception

AWS Kinesis Firehose allows Traverse to send you large amounts of data without the need for you to run a webserver to receive this data. The Firehose writes files to S3 and you can trigger jobs on an interval or using AWS Lambda and S3 Triggers to process these files. This documentation page covers using AWS CloudFormation setting up an IAM Role for the Firehose, setting up the Firehose, and setting up an IAM Role that Traverse can STS Assume to write to the created Firehose.

Firehose Infrastructure Creation

The following Cloudformation templates should be run in-order, with outputs from one being passed to the next.

1. Firehose IAM Role

Set up an IAM Role that Firehoses can use to write to S3. You should already have a S3 Bucket created that you will be writing to.

Parameters

Parameter Description
s3BucketArn The ARN of the S3 Bucket (created) your firehose will write to. Should be something like: arn:aws:s3:::my-bucket-name

Cloudformation Template

kinesis_firehose_iam_role.yml

AWSTemplateFormatVersion: 2010-09-09
Description: IAM Role for a running Kinesis Firehose

Parameters:
  s3BucketArn:
    Type: String
    AllowedPattern: ".+"
    Description: The s3 bucket to push to

Resources:
  KinesisFirehoseIamRole:
    Type: "AWS::IAM::Role"
    Properties:
      Path: /kinesis/
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service:
                - "firehose.amazonaws.com"
            Action:
              - "sts:AssumeRole"
      Policies:
        - PolicyName: FirehosePolicy
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: "Allow"
                Action:
                  - "s3:*"
                Resource:
                  - Ref: s3BucketArn
                  - "Fn::Join":
                      - ""
                      -
                        - Ref: s3BucketArn
                        - "/*"

Outputs:
  kinesisFirehoseIamRole:
    Description: "Resulting IAM Role Arn for the Kinesis Firehose"
    Value: !GetAtt [ 'KinesisFirehoseIamRole', 'Arn' ]

2. Firehose

Set up the firehose, which will use the kinesisFirehoseIamRole output of the kinesis_firehose_iam_role.yml template as the firehoseArn.

Parameters

Parameter Description
s3BucketArn The ARN of the S3 Bucket (created) your firehose will write to. Should be something like: arn:aws:s3:::my-bucket-name
s3Prefix The path to which the Firehosed files will be written to. Something like traverseEvents/
firehoseArn The kinesisFirehoseIamRole output of the kinesis_firehose_iam_role.yml template
bufferingInterval The interval of buffering in seconds
bufferingSize The maximum buffering size in megabytes

Cloudformation

kinesis_firehose_s3_write.yml

AWSTemplateFormatVersion: 2010-09-09
Description: A firehose steam
Parameters:
  s3BucketArn:
    Type: String
    AllowedPattern: ".+"
    Description: The s3 bucket to push to
  s3Prefix:
    Type: String
    Default: "events/"
    AllowedPattern: ".+"
    Description: The path in the s3 bucket to push to
  firehoseArn:
    Type: String
    AllowedPattern: ".+"
    Description: The arn the firehose assumes
  bufferingInterval:
    Type: Number
    Default: 60
    MinValue: 60
    MaxValue: 900
    Description: the interval of buffering in seconds
  bufferingSize:
    Type: Number
    Default: 50
    MinValue: 10
    MaxValue: 128
    Description: the maximum buffering size in megabytes

Resources:
  KinesisFireHose:
    Type: "AWS::KinesisFirehose::DeliveryStream"
    Properties:
      S3DestinationConfiguration:
        BucketARN:
          Ref: s3BucketArn
        BufferingHints:
          IntervalInSeconds:
            Ref: bufferingInterval
          SizeInMBs:
            Ref: bufferingSize
        CompressionFormat: UNCOMPRESSED
        Prefix:
          Ref: s3Prefix
        RoleARN:
          Ref: firehoseArn

Outputs:
  firehose:
    Description: "Resulting Name of the firehose"
    Value: !Ref 'KinesisFireHose'
  firehoseArn:
    Description: "Resulting ARN of the firehose"
    Value: !GetAtt [ 'KinesisFireHose', 'Arn' ]

3. Traverse IAM Role for Firehose Writing

Set up the role that Traverse will STS Assume to write to the created Firehose.

Parameters

Parameter Description
externalRoleToPermission The ARN of traverse’s instance that will do the assuming. The default value is the right IAM Role to use.
firehoseArn The firehoseArn output of the kinesis_firehose_s3_write.yml template. You should only need to specify this.
rolePath A path prefix for the IAM Role. If you keep your IAM Roles organized, you might want to set this to your schema.

Cloudformation

kinesis_firehose_assume_role.yml

AWSTemplateFormatVersion: 2010-09-09
Description: IAM Role that an external instance can assume to push to an internal Kinesis Firehose
Parameters:
  externalRoleToPermission:
    Type: String
    Default: "arn:aws:iam::558989422793:role/app/ApplicationIamRoles-SqsWorkerIamRole-9CRD1C8WFRFQ"
    Description: The arn of the iam role to permission to allow assumption of this new role
  firehoseArn:
    Type: String
    Description: The arn of the firehose instance.
  rolePath:
    Type: String
    Default: /external/traverse/
    Description: The path of the iam role to create


Resources:
  KinesisFireHosePusherIamRole:
    Type: "AWS::IAM::Role"
    Properties:
      Path:
        Ref: rolePath
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              AWS:
                Ref: externalRoleToPermission
            Action:
              - "sts:AssumeRole"
      Policies:
        - PolicyName: KinesisFirehosePusherPolicy
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: "Allow"
                Action:
                  - "firehose:PutRecord"
                  - "firehose:PutRecordBatch"
                Resource:
                  Ref: firehoseArn

Outputs:
  iamArn:
    Description: "Resulting IAM Role Arn that can be assumed"
    Value: !GetAtt [ 'KinesisFireHosePusherIamRole', 'Arn' ]

Traverse Details

Traverse needs to know which IAM Role to assume and the name of the Firehose to write to. Please send the following back to your Traverse representative:

  1. The firehose output from the Firehose (kinesis_firehose_s3_write.yml) template.
  2. The iamArn output from the Assume IAM Role (kinesis_firehose_assume_role.yml) template.