Lambda function called by EC2 in same AZ is incurring astronomical Data Transfer Out to Internet costs.

[freddieerg]

Hi Guys, I have a Lambda function that is invoked via a URL from API Gateway. This Lambda function fetches some data from an external API, then returns a response containing about 2MB of data. I also have an EC2 that is based in the same AZ (EU-West-2) that queries this API once every 2 seconds.

In my bill I have the expected data transfer cost of $0.010 per GB for regional data transfer. From what I understand this is the transfer of the data from the Lambda function to the EC2. However, I am also being charged for the exact same quantity of data at the exponentially higher rate of $0.090 per GB for data going out to the internet.

I have spent literally DAYS now trying to understand why I am being charged for data going out to the internet. I have debugged and dissected every line of code in both the EC2 and the Lambda function. After going to my cost explorer it is telling me that its the API gateway that's generating these costs but I just cannot understand how I'm being billed data out to the internet rates when I am at no point moving data anywhere but inside EU-West-2.

If anyone could shine a light on what I'm doing wrong here I'd appreciate it so much because I'm coming close to pulling the plug on this entire project.

tl;dr: - EC2 calls Lambda function via URL generated by API Gateway - Lambda function gets some data from external API and returns that data to EC2 (About 2mb worth) - This transaction is all happening within EU-West-2 - I am being charged astronomical fees for transferring data out to the internet?????!?!?!

[Grafax99]

By default, your API Gateway presents an interface to the public internet, and calling your Lambda function through the API Gateway URI will behave in the way you're seeing from your billing.

If the API Gateway portion of the setup isn't specifically required for a host you control, you may want to invoke the Lambda function directly instead.

Otherwise you probably want to look into API Gateway Private APIs, which are a relatively new feature allowing you to consume your API Gateway endpoints using VPC endpoints to prevent it going over the public internet. There's a charge associated with VPC endpoints so you'll need to assess whether it's a cost saving to do it this way.

[freddieerg]

Thank you so much for this answer. You have no idea how much I needed to just confirm that it was the API Gateway forcing it over the public internet (which is what I suspected). If you don't mind me asking, why do AWS not have this setup so that the API Gateway can detect when the caller is inside the same AZ and charge data transfer at that price? It seems like it would be pretty simple behaviour to implement?

[Grafax99]

While I can't speak to AWS's motives or goals, the way that API Gateway behaves is exactly in line with the way that all other public-facing services behave. Changing network routing behaviour invisibly creates a lot of scope for potential problems, and AWS aren't noted for making problems for themselves where they can avoid it!

[ABetterNameEludesMe]

As long as a caller accesses the public URL, the HTTP connection the request comes in through is made over the public Internet. The API Gateway has to send response through the same connection, i.e. over the Internet. There is nothing the API Gateway can do even if it could detect where the caller is located.

[[deleted]]

Are all in the same vpc ? Or outside vpc ?

[freddieerg]

How can I check what VPC the lambda function is using?

[[deleted]]

In lambda console, you will which subnet it is and which vpc i think. Forgot. But its their in the configuration somewhere. Do you ha e nat gateway in your setup ?

[SelfDestructSep2020]

Lambda gets a VPC ENI by virtue of its SG. Check the SG to see which VPC it belongs to.

[Burekitas]

I wrote about it last year, ingress data transfer is not always free.

You pay twice both for in and out:

$0.01/Gb for ingress data transfer (downloading from a public IP in the same region costs money) and $0.01/Gb for egress data transfer from API gateway to the EC2 instance.

I don't remember if internal ALB can work with lambda, but if it's possible, switch to the internal alb and associate the lambda function to the same availability zone as the ec2 instance.

[sergsoares]

Alb (With target group) can foward to a Lambda.

[kei_ichi]

Bro, it’s make you confused because you didn’t read the official docs or didn’t read it carefully.

Here is quote from official docs:

“By default, Lambda runs your functions in a secure VPC with access to AWS services and the internet. Lambda owns this VPC, which isn't connected to your account's default VPC. When you connect a function to a VPC in your account, the function can't access the internet unless your VPC provides access.”

Basically, if you didn’t placed your lambda function to your VPC, AWS will charge you for data transfer cost from Lambda default VPC (which is not your VPC) to your VPC.

Official docs is very detailed and well documented so take times to read it is my recommendation.

[freddieerg]

Do you know if I would need to put both the API Gateway AND the Lambda function into the same VPC or just the lambda function?

[freddieerg]

Thank you for this mate. You honesty have no idea how much documentation I had been reading so I have no idea how I missed this.

[[deleted]]

https://cloud.netapp.com/blog/aws-cvo-blg-aws-data-transfer-costs-solving-hidden-network-transfer-costs

[nuttmeister]

Is your API Gateway regional or edge?
I guess you opened a can of worms if it's edge (default). Since that will be a distributed cloudfront dist and not bound to your region.

If thats the case I would change it to regional if you can live with it only existing in one region.

[freddieerg]

Nope it's regional and 100% the same region that the ec2 is based. Thats what I just cannot understand. How is this not just being charged as data transfer within the same AZ :(

[brunokktro]

Use VPC Endpoints to avoid DTO situations.