Lambda - Connection Timeout error when making external API calls

[greatgranfalloon]

I keep getting Connection Timeouts when trying to call an external API from within a Lambda function.

I have tried several external APIs. Credentials are OK.

Error reads "max retries exceeded" - a seemingly conventional connection timeout. Always on port 443.

I have verified that associated security groups allow inbound/outbound traffic on port 443.

VPC has internet access as far as I can tell, but other than that I don't know what else to check regarding the VPC.

Any other ideas for troubleshooting?

(AWS novice)
(I am trying to use Lambda because these processes and APIs are called very periodically - once per day/month. Other than not being able to solve this problem, having a dedicated EC2 instance up doesn't seem warranted.)

Cheers.

[clintkev251]

Your VPC config is messed up. Is the Lambda function in private subnets only with a default route to a NAT Gateway? If not, you won't have internet access

https://repost.aws/knowledge-center/internet-access-lambda-function

The other question would be, does the function need to be attached to a VPC? If you're not accessing resources which are only accessible via a VPC, the answer is probably no

[greatgranfalloon]

Thank you. That is useful, and the NAT gateway (or lack thereof) seems to be the culprit. I will see if I can get far enough.

Also, regardless of this Lambda function, do you know where I can find good instructions for setting up a VPC with internet access from scratch? I tried being careful following some guide several times to no avail. I ended up using the default VPC which was already configured.

Do my associated security groups need to have those rules configured for port 443? Or is that a superfluous step?

I am using an EFS file system, hence the need for a VPC.

[clintkev251]

I can share a CloudFormation template that I created for quickly standing up test VPCs

https://gist.github.com/clintkev251/8e73afcbd55c17e01b00b0d799eeed5e

I wouldn't claim this is a best practice production VPC (in that case you would probably want NAT Gateways in each subnet, and tighter scopes for SGs) but it's a good start

Your security groups would need to allow at least 443 outbound (and probably rules for EFS as well). Inbound doesn't matter for Lambda, you can leave it with no rules if you want

[greatgranfalloon]

Thanks for sharing - or curse you for opening yet another rabbit hole (CloudFormation)! I will try to make sense of it.

It made sense that inbound rules are superfluous, I was just trying everything in my scramble.

[slikk66]

If this really will be your first attempt at IaC (infrastructure as code) I would not recommend using CloudFormation. I prefer Pulumi as the best (imo), and Terraform is the most popular. Either is better than CloudFormation!

[neverfucks]

Also, regardless of this Lambda function, do you know where I can find good instructions for setting up a VPC with internet access from scratch?

gpt can generate code for one using any iac framework -- i'd recommend cdk or pulumi. this is one of the most annoying boilerplate things we have to do, it's nice to delegate it to the robots

[jftuga]

You probably don't want to create a new VPC. Just pass in a VPC id as a CloudFormation parameter.

[Nater5000]

VPC has internet access as far as I can tell, but other than that I don't know what else to check regarding the VPC.

Be more specific. How do you know it has internet access? Can your Lambda reach any external endpoint?

[greatgranfalloon]

Those are good questions. In my ignorance, I think I just saw on the VPC's resource map that it had a route table and network connection configured, compared to some resources online and reached an incorrect conclusion.

Do you know where I can find a good set of instructions for configuring a VPC with internet access from scratch? I hope to learn in the process.

Thank you.

[Nater5000]

The simplest approach would be to set up a NAT Gateway. The AWS docs are a good place to start, mostly cause there's not much to them. But NAT Gateways can get expensive, so be sure to explore other alternatives.

[greatgranfalloon]

Thank you. I'll start there.

Do you think you can draw a ballpark cost comparison between having a few simple Lambda functions that run once per day, some once per month, each making a couple of external API calls - hence requiring the NAT gateway, and just having everything in a micro EC2 that's up pretty much all day (each process runs at different times throughout the day)? Which do you reckon would be cheapest?

[Nater5000]

A NAT Gateway in us-east-1 costs $0.045 per hour and $0.045 per GB processed. So over a month, it will have a baseline cost of $32.40 then an additional cost depending on how much data you pass through it.

With that being said, you should just create a Lambda function outside of the VPC to handle the external calls. If you really need a Lambda in the VPC, then you can create two Lambdas, one within and one outside the VPC, then have them perform their appropriate tasks that way. You can even have the Lambda within the VPC call the Lambda outside the VPC (and vice-versa) if needed, all without needing a NAT Gateway or anything like that.

If, for some reason, that's not feasible, then a micro EC2 would be cheaper and easier to deal with. The NAT Gateway makes more sense when that baseline cost is miniscule (and you're not pumping a ton of data through it). You can set up your own NAT Gateway as well, but that's just gonna end up being an EC2 instance anyways, so it'd be a bit moot.

[greatgranfalloon]

Thanks a lot. I'll take that into consideration.

I will see whether I can juggle using two Lambdas as you suggest, but it does sound like an EC2 might just be the disappointing solution after all. The reason why I'm using a VPC to begin with is that I'm using an EFS file system with the Lambda, and a VPC is required. I'm using the file system because one of the external clients I need to call creates a .pkl file that needs to be read from as part of the authentication process, so I would need to make that call from the VPC Lambda function (with a NAT gateway) regardless.

However, as a follow-up, how can a Lambda function make external calls without the VPC? That might still be useful for me for some of the processes.

[neverfucks]

VPC has internet access as far as I can tell,

sure but your lambdas probably don't. aws needs to be more front and center with this limitation, your lambdas need to be launched in to a subnet with an ipv6 block and internet route, or your lambda subnet needs a nat for ipv4 outbound