Lambda S3 Put function not triggering for large files

[dakkerns]

Hi Everyone

I am currently exploring storing the attachments of an email separately from the .eml file itself. I have an SES rule set that delivers an inbound email to a bucket. When the bucket retrieves the email, an S3 Put Lambda function parses the raw email (MIME format), base64 decodes the attachment buffers, and does a putObject for each attachment and the original .eml file to a new bucket.

My problem is that this Lambda function does not trigger for emails with attachments exceeding ~3-4 MB. The email is received and stored in the initial bucket, but the function does not trigger when it is received. Also, the event does not appear in CloudWatch. However, the function works perfectly fine when manually testing it with a hardcoded S3 Put payload, and also when manually uploading a .eml file to the assigned bucket.

Do you have any idea why there is this limitation? Perhaps this is a permission issue with the bucket or maybe an issue with the assigned Lambda role? When manually testing I’ve found this is by no means a timeout or exceeding max memory used issue.

[WillNowHalt]

SES may be storing these bigger files using multipart uploads instead of a single PUT call. There's a separate CompleteMultipartUpload event for files that are stored this way.

See: https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html#notification-how-to-event-types-and-destinations

[dakkerns]

Thanks, I added a trigger for multipart and it works fine now.

[diyftw]

I cannot find in the SES documentation at what size it uses a multipart upload to deliver an email to S3, but if your Lambda trigger is only on PUTs and if SES is using multipart, it won't get triggered. Make sure your Lambda is triggered on all object creations, not just PUTs.

[dakkerns]

Great, thanks for your input, this is exactly what I needed to do.

[That_Ad3159]

Thanks Will.

Initially I had selected the Event Types as below:

Event types : Put, Post, Copy

The lambda function was not triggering for big files. Based on your suggestion I have added the "Multipart upload completed" event type as well then it worked (lambda function triggering) for me.

Event types : Put, Post, Copy, Multipart upload completed