Getting around circular CloudFormation Dependencies: S3-Event-Lambda-Role

Thumbnail

Getting around circular CloudFormation dependencies

Several posts complain about the inability of CloudFormation to apply a Lambda event function to an S3 Bucket with an dynamically generated name.

The standard UseCase is an S3 Bucket with a Lambda event notification. In this special case the Bucket has a dynamically generated name. This cannot be done by pure CloudFormation!

How to work around this circular depency? Let me show you an easy way:

In AWS Blog: Handling circular dependency errors in AWS CloudFormation [AWS1] and serverless hero Ben Kehoe writes here .[BEN1]

This will not work

So in pure CloudFormation this is not possible:

BucketEvent

That is because we have a circular dependency. As [AWS1] says: “The bucket notification is dependent on the Lambda function and the Lambda function is dependent on the execution role, which is dependent on the S3 bucket.”

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
  Bucket:
    Type: AWS::S3::Bucket     
  Function:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: s3://bucketname/object.zip # Add in an S3 URI where you have code Lambda Code
      Runtime: python2.7
      Handler: index.handler
      Policies:
        - Version: 2012-10-17
          Statement: 
            - Effect: Allow
              Action: s3:GetObject*
              Resource: !Sub "arn:aws:s3:::${Bucket}*"
      Events:
        Upload:
          Properties:
            Bucket:
              Ref: Bucket
            Events: s3:ObjectCreated:*
          Type: S3

So we have to decouple things. We can do that by first creating the bucket and the lambda function and bringing them together after that.

The way to extend CloudFormation is a custom resource. That is a Lambda Function wich acts as a CloudFormation resource. This “bring together” function gets notified to create or update or delete itself.

You could also do this as an AWS cli script after creation of the CloudFormation stack. But if you do that, you won’t have the stack itself as a whole functional unit. But beeing lazy, the question is: do I really have to code this all by myself? - Answer is no - CDK to the rescue.

This will work

BucketEvent

CDK code

If you create an CDK app with cdk init app --language=typescript you just have to get some imports:

import {Bucket,EventType} from '@aws-cdk/aws-s3';
import {LambdaDestination} from '@aws-cdk/aws-s3-notifications';
import {Function, Runtime, Code} from '@aws-cdk/aws-lambda'

And replace // The code that defines your stack goes here with this snippet:


export class CfnGraphStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);


    // S3
    const bucket = new Bucket(this, "testbucketmpa",{
      removalPolicy: cdk.RemovalPolicy.DESTROY, // NOT recommended for production code
    });

     // Lambda
    const lambda = new Function(this, 'HelloHandler', {
      code: Code.asset(path.join(__dirname,  '../lambda')),
      handler: 'hello.handler',
      runtime: Runtime.NODEJS_8_10,
      memorySize: 1024
    });

    // S3 -> Lambda
    bucket.addEventNotification(EventType.OBJECT_CREATED, new LambdaDestination(lambda));


  }
}

Of course there have to be a node Lambda function in the dir lambda

The CDK has already added the “bring together” Lambda function. And it is all happening under the hood, we just add the EventNotifcation in the ts code to the Bucket.

Generated template

 testbucketmpaNotificationsB2722499:
    Type: Custom::S3BucketNotifications
    Properties:
  ...
      BucketName:
        Ref: testbucketmpaAE8E5392
      NotificationConfiguration:
        LambdaFunctionConfigurations:
          - Events:
              - s3:ObjectCreated:*
            LambdaFunctionArn:
              Fn::GetAtt:
                - HelloHandler2E4FBA4D
                - Arn
    DependsOn:
      - HelloHandlerAllowBucketNotificationsFromCfnGraphStacktestbucketmpa0C1D72A46851358B

This snippet from the generated CloudFormation (cdk synth) shows the Custom Ressource which waits (with DependsOn until the AWS::Lambda::Permission is created.)

The Lambda Function is created inline:

BucketNotificationsHandler050a0587b7544547bf325f094a3db8347ECC3691:
    Type: AWS::Lambda::Function
    Properties:
      Description: AWS CloudFormation handler for "Custom::S3BucketNotifications" resources (@aws-cdk/aws-s3)
      Code:
        ZipFile: >-
          exports.handler = (event, context) => {
              const s3 = new (require('aws-sdk').S3)();
              const https = require("https");
              const url = require("url");
              log(JSON.stringify(event, undefined, 2));
...
              return s3.putBucketNotificationConfiguration(req, (err, data) => {

Its creates the BucketNotificatioConfiguration and you’re done!