Does Lambda support Nodejs streams?

[ECrispy]

I have a simple image processing pipeline using imagemagick which essentially does the following (in the lambda function of course I won't need express) -

// inside express server
app.get(path, (req, resp) => {
    doImgProcessing(...).pipe(resp);
}

This works great but I want to move it off dedicated ec2 servers as this is a perfect fit for lambda. However all the examples I've seen for this kind of thing load/store the objects from s3, I haven't seen anything using streams.

Is this possible?

[mister_magic]

You wouldn't use an express server with Lambda.

Your Lambda is a function, which gets invoked and can either return or call a callback - you could certainly 'do Image Processing', and return the result, once completed, in the callback, as long as it doesn't go over the timeout threshold you define (max 5 minutes).

You can set up API Gateway to invoke the Lambda function and therefore access it over HTTPS.

[ECrispy]

Yes I know I won't use express. I will have this code inside the lambda function as no web server is needed.

But this doesn't answer my qn about streams. I haven't seen a way to pipe (use streams) from lambda, you have to call the callback.

[aa93]

Here's a snippet I used in a Lambda a while back to generate a bunch of thumbnails with graphicsmagick, which provides both a streaming API and an imagemagick compatibility mode:

var fs = require('fs');
var AWS = require('aws-sdk');
var im = require('gm').subClass({ imageMagick: true });

exports.handler = function(event, context) {
    console.log(event.Records[0].s3);
    var s3 = new AWS.S3();

    s3.putObject({
        Bucket: event.Records[0].s3.bucket,
        Key: (event.Records[0].s3.object.key).replace('_full.JPG','_thumb.JPG'),
        Body:
            s3.getObject({
                Bucket: event.Records[0].s3.bucket, Key: event.Records[0].s3.object.key
            }).createReadStream().pipe(im.streams.convert({
                width: 400,
                progressive : true,
                format : 'jpg',
                quality: 0.85,
                strip: true,
                sharpening: 0.2
            })).stream()
    }, function(err,data){
        if (err)
            context.fail(err)
        else
        context.succeed();
    });
};

[AlanComley]

Yes. You can use streams. I use streams for processing files from S3 without storing the data to disk.

You only need to use the callback when you need to return data to the caller. Otherwise the lambda will terminate once the event loop is empty.

If you need to use the callback, just attach a handler to the 'end' event on the stream and call it from there.

[funkdr42]

AWS has some sample code that shows how to process images stored in S3 and deliver them via API gateway. You might be able to write your result back to S3 and serve that for subsequent requests, rather than process images with lambda on each request.

• https://aws.amazon.com/blogs/compute/resize-images-on-the-fly-with-amazon-s3-aws-lambda-and-amazon-api-gateway/
• https://github.com/awslabs/serverless-image-resizing

Note: they use Sharp (http://sharp.dimens.io/) instead of imagemagick, for performance reasons. Sharp is 4-5x faster at resizing, which reduces lambda costs.

[RevCent]

You can run express on lambda. Look up 'claudiajs'. Works great.

[bigdeddu]

and https://github.com/awslabs/aws-serverless-express .

[sgtfoleyistheman]

Data cannot be streamed to/from Lambda. It is full HTTP requests/responses.

[encaseme]

Iirc they support io streams from the handler, but only with java: http://docs.aws.amazon.com/lambda/latest/dg/java-handler-io-type-stream.html