The case of the missing bucket notifications

Thumbnail

The case of the missing bucket notifications

A few days ago I was trying to do something quite simple. I wanted to send S3 Put-Events to multiple Lambda functions for processing. This is a pretty common pattern. To implement it you have to use an architecture such as the one you can see in the title image above.

This is because S3 has a limit on the event handlers (notification targets) per event type of exactly one. That’s not a lot. To work around that it’s common to use SNS as an intermediary to fan out the event notifications to multiple targets.

I had done the same as part of a larger infrastructure but for some reason my lambdas weren’t getting any events from S3.

I tried a lot to figure out the issue, but everything seemed to be configured correctly.

  • The deployment pipeline was deploying my CloudFormation template correctly
  • The notification was configured correctly in the S3 GUI.
  • Even after changing the name of the notification in the GUI the S3-check still reported it as being configured correctly
  • Publishing other events to SNS triggered my Lambdas

So it seemed like the problem was the communication between S3 and SNS. That was bad, because there’s not a lot of logs (none) you can look at.

I built the whole thing manually in my personal account via the GUI and everything seemed fine - all notifications were being triggered as expected. That annoyed me a little to say the least so I finally decided to extract the involved parts from the infrastructure and create a minimal CloudFormation template for it.

Resources:
​
    EncryptionKey:
        Type: AWS::KMS::Key
        Properties:
          Description: KMS Key
          KeyPolicy:
            # Allow access from our AWS Account
            Version: "2012-10-17"
            Id: "kms-key"
            Statement:
              - Sid: "Enable IAM User Permissions"
                Effect: Allow
                Principal:
                  AWS:
                    Fn::Join:
                      - ""
                      - - "arn:aws:iam::"
                        - !Ref AWS::AccountId
                        - ":root"
                Action: "kms:*"
                Resource: "*"
​
    S3BucketFileStorageBucket:
        DependsOn: NewFileNotifications
        Type: AWS::S3::Bucket
        Properties:
          BucketName: randombucketname0928439843293874
          # Enable Encryption
          BucketEncryption:
            ServerSideEncryptionConfiguration:
              - ServerSideEncryptionByDefault:
                  SSEAlgorithm: AES256
          # Block all public access
          AccessControl: Private
          NotificationConfiguration:
            TopicConfigurations:
              - Event: "s3:ObjectCreated:*"
                Topic: !Ref NewFileNotifications
          PublicAccessBlockConfiguration:
            BlockPublicAcls: true
            BlockPublicPolicy: true
            IgnorePublicAcls: true
            RestrictPublicBuckets: true
​
    NewFileNotifications:
        Type: AWS::SNS::Topic
        Properties:
          KmsMasterKeyId: !Ref EncryptionKey
          TopicName: NewFileNotifications
​
    NewFileNotificationsTopicPolicy:
        Type: AWS::SNS::TopicPolicy
        Properties:
          PolicyDocument:
            Id: AllowPublishFromS3
            Version: '2012-10-17'
            Statement:
              - Sid: Statement-id
                Effect: Allow
                Principal:
                  AWS: "*"
                Action: sns:Publish
                Resource:
                  Ref: NewFileNotifications
                Condition:
                  ArnLike:
                    aws:SourceArn: 'arn:aws:s3:::randombucketname0928439843293874'
          Topics:
            - Ref: NewFileNotifications

Minimal insofar as CloudFormation templates can be minimal. I deployed it and saw the same behaviour as in the real environment, which was good… and bad. It meant I had done something wrong and couldn’t blame AWS.

You might have noticed something I hadn’t mentioned yet - the bucket and topic were encrypted via KMS. Turns out looking at my own code would have helped - my comment mentions Allow access from our AWS Account but the statement Id says Enable IAM User Permissions. Past me had lied and present me believed him and assumed the account hat access to the key.

This section allows all IAM-Users to access the key:

    EncryptionKey:
        Type: AWS::KMS::Key
        Properties:
          Description: KMS Key
          KeyPolicy:
            # Allow access from our AWS Account
            Version: "2012-10-17"
            Id: "kms-key"
            Statement:
              - Sid: "Enable IAM User Permissions"
                Effect: Allow
                Principal:
                  AWS:
                    Fn::Join:
                      - ""
                      - - "arn:aws:iam::"
                        - !Ref AWS::AccountId
                        - ":root"
                Action: "kms:*"
                Resource: "*"

That’s why the checks S3 does to verify the connection to SNS worked fine when I executed them from the GUI and during CloudFormation deployment. Both are triggered via real Users - in the case of the GUI my own user and the technical user for the pipeline.

After finding out what I had to look for, I quickly found this AWS blog, which describes how to do it properly. Once you know what’s wrong, it seems like the whole internet tells you how to do it properly and makes you feel like an idiot for not knowing it.

After some more research I changed the encryption key configuration to this:

Resources:
​
    EncryptionKey:
        Type: AWS::KMS::Key
        Properties:
          Description: KMS Key
          KeyPolicy:
            # Allow access from our AWS Account
            Version: "2012-10-17"
            Id: "kms-key"
            Statement:
              - Sid: "Enable IAM User Permissions"
                Effect: Allow
                Principal:
                  AWS:
                    Fn::Join:
                      - ""
                      - - "arn:aws:iam::"
                        - !Ref AWS::AccountId
                        - ":root"
                Action: "kms:*"
                Resource: "*"
              - Sid: "Allow S3 to encrypt"
                Effect: Allow
                Principal:
                  Service: s3.amazonaws.com
                Action:
                 - "kms:GenerateDataKey*"
                 - "kms:Decrypt"
                Resource: "*"
              - Sid: "Allow SNS to encrypt"
                Effect: Allow
                Principal:
                  Service: sns.amazonaws.com
                Action:
                 - "kms:GenerateDataKey*"
                 - "kms:Decrypt"
                Resource: "*"

It allows SNS and S3 to use the KMS Key to encrypt messages.

Summary

If S3 notifications to SNS fail silently, check if you have encryption enabled and make sure that SNS and S3 have access to the key. Out of shame I won’t disclose how long it took me to figure this out ;-)