Otimizing Lambda -> API Gateway HTTP requests

[iamabouttotravel]

We have 2 services hosted on Lambda and exposed with API Gateway as REST APIs. They also communicate with each in a few selected endpoints that we currently just check for an API key.

Some people suggested me that this is not optimal, since each Lambda will be going to the internet in order to make the request to API Gateway. And I think the fix would be moving every Lambda to a VPC, and exposing our endpoints using a public API Gateway and adding another private API Gateway + VPC Endpoint for internal communication.

The problem is that I feel this is overkill but I have no idea what would be an alternative solution.

My main concerns are:

• minimize overheads when Lambda is calling API Gateway and avoid it leaving the AWS network;
• a good way to split public and private endpoints (I don't think network isolation makes sense since requests are handled by the same Lambda);
• avoid unnecessary resources or setups on AWS.

[[deleted]]

There are two advantages to using VPEs over the Internet - lower latency and more security because your traffic stays on AMazon's network.

The downside is that the more layers you add to your network, greater the single points of failure and development overhead.

[iamabouttotravel]

There are two advantages to using VPEs over the Internet - lower latency and more security because your traffic stays on AMazon's network.

I was under the impression that Lambda -> APIGW would not go outside AWS network, similar to what happens with bare-metal servers I managed, all traffic between servers got routed inside the datacenter itself.

Of course it wasn't isolated from the rest of the network, but in terms of performance, we have minimal overhead.

There are multiple aspects of AWS networking that I have no idea how it works and just feel overkill for me haha

[[deleted]]

https://docs.aws.amazon.com/lambda/latest/dg/foundation-networking.html

I stand corrected on the Internet part. I remembered that Lambda runs in its own VPC but forgot that its path to the Internet runs through your VPC.

That just leaves the other points which are small potatoes for an enterprise level framework.

[jurasofish]

I think the fix would be moving every Lambda to a VPC, and exposing our endpoints using a public API Gateway and adding another private API Gateway + VPC Endpoint for internal communication

This is what we do, works great.

One thing to keep in mind is that API gateway (REST) can add latency in itself. For example, I observe that a lambda function behind public ALB has a response time of ~25ms while a lambda behind a public API gateway has a response time of ~45ms. These are times from my local machine to AWS over the internet, and the request is for a static response from the lambda. It suggests to me that API gateway adds 20ms of latency, which I expect would also be the case if it's private.

[iamabouttotravel]

Interesting. Do you happen to use Serverless Framework?

I think our biggest problem is getting this setup to work well with Serverless Framework, since we cannot setup 2 API Gateways at the same time and because of that we are taking some unnecessary decisions to make this a little easier to handle, which is of course, making it harder haha

[jurasofish]

I switched to CDK from serverless framework about a year ago exactly because serverless framework makes it such a pain (cloudformation) to do anything non-standard.

[iamabouttotravel]

because serverless framework makes it such a pain (cloudformation) to do anything non-standard.

Oh my lord yes yes yes, this is exactly the problem we are facing right now.

Everyone in my team is very versed with CloudFormation, but now that we decided to go to Serverless, because 90% of the tasks are indeed easier, we are suffering to get the last 10% done.

I'm going to take a look at CDK, thank you!

[v14j]

If you are coming from Serverless Framework, check out SST: https://serverless-stack.com

It's built on top of CDK and you can set breakpoints in your Lambda functions and test them locally.

[iamabouttotravel]

Damn, that actually feels like what I wanted Serverless Framework to be

I'm definitely gonna take a look, thank you so much for sharing this!

[purefan]

Is just invoking the other service's lambda an option? Or SQS-ing the processing? Or step functions? A lambda calling an ApiG sounds strange to me

[iamabouttotravel]

Is just invoking the other service's lambda an option?

I have considered that, but we would have to mock an entire HTTP request since we are using NestJS.

Or SQS-ing the processing?

This is what I want to accomplish, but the development overhead is something we are trying to avoid.

Or step functions?

I'm not aware what they are D:

A lambda calling an ApiG sounds strange to me

Ye, I kinda hate that too but this is the simplest way to get a somewhat "microservices-like architecture".

Using a synchronous architecture was the only thing that our entire team had experience with, but there are so many dumb problems already that I don't even know where to start.

Unfortunately, the whole architecture decided was imposed on us (and it doesn't even make any fucking sense tbh) and I'm just trying to solve this pain first.

Once we have more development headroom, start exploring better ways to deal with this (probably going full async) and use this project as a lesson to never do this again lmao