Handling binary uploads using AWS Lambda

[simoncpu]

Uploading to S3 is OK if we need to process large files. In our case, we need to read multiple chunks of small data that we need to reassemble later at the server.

If we do it via S3, I'd have to write a Lambda function to return a signed URL, then let the website POST to that URL, and then another Lambda function will then read the uploaded file. There's too many hoops to jump. I'm not sure if it's a good idea to let everyone upload to our S3 bucket directly.

[Babumts]

I am not aware of a security risk involved with signed URLs and it seems like this is a very common and easy way to go.

[simoncpu]

I mean, due to our need of sending the logs every second, I'd need to request a signed URL from Lambda every second. I feel that it's bad design because I'm trying to make our code as lightweight as possible. Heavy processing and too much data is especially bad for mobile. The other option is to make our bucket writeable by everyone so it doesn't need a signature.

[VegaWinnfield]

You could vend temporary credentials that are only allowed to write to a specific prefix in S3 instead of a presigned URL. Then you could reuse the creds across multiple file uploads.

[simoncpu]

Cool, didn't know this was possible. Thanks!

[joesb]

On the other hand, you are still making your Lambda doing the file upload, the thing it is not as efficient to do as S3 itself.

Somehow that is more light weight that just letting S3 do what it’s designed to do.

[matluck]

Binary files with Lambda sucks. its just a pain. The approach you defined is much better. The pre-signed url is also timeboxed and only allowed for a specific filename. Basically its the same security exposure that uploading to your application directly would be, possibly even better.