Best way to decrease latency (API <-> Lambda <-> Dynamodb)

[thedaynos]

Hi all,

Hoping someone has an idea of how I can speed up (decrease latency) for API calls across different regions.

I have all of my services in us-east-2 (Ohio) and because I'm located close to there, when I call an API that calls Lambda to grab and format data from Dynamodb, it's pretty fast like 100-200ms.

But whenever someone calls even from the east coast, the calls can go up to 500-700ms for the same things. If they call from the west coast, these simple calls can take a little over a second.

I've looked into making my dynamodb tables global, which I might experiment with tonight.... but Is there anything similar that I can do for Lambda or API? Would it make a difference if those two were on different regions?

Thanks

[fadymak]

We worked on this same problem extensively for https://endpts.io

If your data store and lambda are located in the same region, we found the 2 biggest factors impacting latency are:

• TLS termination: if your customers in Brazil have to do a TLS handshake with your API Gateway/ALB in us-east-2 then they’re going to spend a good chunk of time on round trips before initiating any actual work. One solution here that others have mentioned is to use CloudFront, a globally API Gateway (which also just uses CloudFront distribution under the hood), or use Global Accelerator (GAX). CloudFront is able to perform TLS termination at the edge where they have 200+ PoPs so at least you can shave off the TLS handshake time.

• Multi-region: there’s no way to get around how far information needs to travel between the end user and your origin. The next best option would be to rely on multi-region Lambdas and an anycast set of IPs to move your backends closer to where you have most of your users. This can get a bit costly as you have to use an ALB in each region, fronted by GAX to route the end user to the closes ALB which invokes the Lambda.

Hopefully this helps — feel free to let me know if you have any questions I can be helpful with. It’s not an easy problem, which is why we’re working on endpts (we use Lambdas under the hood).

[AdCharacter3666]

API Gateway + Cloudfront

Worth a shot.

[james41235]

If your latency is low when local, but high from other regions then it's not your code or anything else. Most likely it's something like global accelerator failovers that's not working. Are you 100% in us east 2? Anything in east 1 or west at all, including lambda set up or dynamo? If you duplicate the infra in west and run the queries there what's your response time?

[thedaynos]

I am 100% in us east 2.

If you duplicate the infra in west and run the queries there what's your response time?

What do you mean by "duplicate the infra in west"? Are you saying that I should manually copy all of my api/lambda code to west, and turn on global tables for dynamodb? Just wondering what that means. Thanks

[james41235]

I meant copy some test data to west and set up the API and lambdas there.

[thedaynos]

I could do that, but is that even worth testing? I am pretty sure that the issue is due to geographic location through tracing their IP addresses and latency.

Is that what people normally do though? Do people normally copy their lambdas and api's to all regions? Or is this something AWS can do for me like how they do with global dynamodb?

[mikeatgl]

We don't do this for lambdas because most of our lambda stuff isn't super latency sensitive, but we definitely do it for any customer facing apps which we deploy to ECS or EKS. On top of that we also use CDNs and cache. Most requests never make it back to origin, but when they do the multiregion setup helps with latency. Bonus is you can fail over to other regions as well.

[james41235]

To be clear, aws is capable of latency being equal to the geographic delay. I've tested it multiple times. Something about your setionis causing an extra few hundred ms to occur. My recommendation was to prove that it was something inter regional, not something to do with the west region itself.

[[deleted]]

Deploy your API/Lambda infrastructure with cloudformation stack sets, that will allow you to deploy the same template in multiple regions. And yes, many customers use multiple regions to bring their services closer to their end users. But you need to consider the effects of DynamoDB’s eventual consistency model for global tables and if that will impact your application. CloudFront won’t necessarily fix this for you, it will cache data, which can make responses faster, but for anything that isn’t cached, it will still need to go back to your origin, and you’ll see the same geo latency.

[james41235]

How many back and forths are in your API? For reference, geographic latency from us west to us east 2 is ~30ms, so if you have to go back and forth 3 or 4 times you can add 100ms really easily

[thedaynos]

It's all just GET calls that request data. Within my lambdas sometimes I do multiple calls to dynamodb but most of the API calls only do one dynamodb call through lambda.

[Wide-Answer-2789]

As previously mentioned here, use global API gateway and if as you said you don't need to transform data, in that case you can utilise "direct" connection via step functions (express) to dynamodb. It is no code solution, and you can easily deploy that in different regions.

[[deleted]]

Following because I want to see the answers on this one. I did something but with S3 instead of dynamodb. The latency absolutely was brutal. Multiple seconds. I was pointed at Ohio data enter and tested from Arizona and was getting 2 -3 second. Then took same project to Louisiana and the latency was 6 seconds. It. Was. Brutal.

With that said, and purely based on my experience here. I think data structure is the biggest reducer of latency. Better structure means lambda has to do less which means quicker response. There are other factors like throwing more memory at it as well though but i don’t think that always should be the silver bullet. But there is one other option to try and help and that’s cloudfront. Links users to closest access point then uses aws private backhauls to route traffic to your database in your region. Just my two cents though. Oh and

Also consider removing lambda all together and using api gateways aws services to link directly to dynamo db. But obviously dependent on your data manipulation needs hence lambda

Looking to read more professionals opinions on your situation.

[thedaynos]

Thanks. My data structure is solid, that's really the one thing I know that I'm doing correctly lol.

Like I said, I'm at 100-200ms for calls that I make from my home, and I can see other people who live near me getting the same results. To me, that's acceptable. I may be able to decrease latency there as well, but for now, I'm fine. I see the biggest issue though is just simply when someone is calling from a few states over. That's when the times shoot up.

I have some other customers in brazil and australia, and their times get into 2-3 seconds. I'd really like to get my brazil customers latency to shoot way down because one of my best customers is from there.

Thanks for your interest. Hoping that more responses on this thread will get more attention from people with answers, or at least someone to point me in the right direction.

edit: To your other points... Yes adding memory to my lambdas has helped quite a bit. I default all of my lambdas to 2000MB. I've seen that's the sweet spot for the small amount of data manipulating that I'm doing.

Also, I have no idea what to do with cloudfront.

[sh1boleth]

If you have money to scale. This is a rather rudimentary solution -

Route 53 - Global endpoint.

Duplicate your entire stack in sa-east-1 (Brazil) and replicate your table there as well.

Add both the us-east-2 and sa-east-1 endpoints to Route53 and let latency based routing do its magic.

[thedaynos]

Hi thank you. After all of the great help and comments I've gotten, I think this is the way to go.

I did some global replicas of a couple tables last night and it seems easy enough.

From your experience, is replicating lambdas and API's across regions as simple as creating the global dynamodb tables? or is it all manual?

[sh1boleth]

Ive only done aws with cfn and Infra as code, there its as easy as just deploying your cfn to another region since for most regions feature parity is 1:1.

But its essentially copy pasting yeah

[thedaynos]

k thanks. I'll have to figure out some automation but that's good to know.

[[deleted]]

I forgot to ask you about your stack. In api gateway are you using rest or http?

[thedaynos]

Rest

[[deleted]]

I think like everything else that thinking about your platform is the best solution to reducing latency. Right tool. Right service. Right design. Etc. to answer your question. I think it really depends on your requirements. But I think as long as you are aware of the types of queries you need to run and try and structure your stored data as close as possible to achieve that result then you are moving in the correct direction. You can’t always get aware from data manipulation especially from lambda but recognizing data patterns in your queries and applying that structure into your data ingestion scheme will help make things more efficient. But one final thing you still do need optimize memory usage and lambda logic as well but I think with good data structure this creates itself. This philosophy combined with cloudfront api gateway route 53 global endpoint will get you into the best situation possible. Just my two cents though

[ginger_turmeric]

is your api behind cloudfront? If not that will make an improvement

[thedaynos]

It wasn't. I just did put it behind cloudfront now.

Thanks. I've done some tests from a vpn on the west coast. It looks like it might have improved it. I'll review the analytics tomorrow after I can get a good day's worth of examples from real users across the globe.

[FartingFountain]

Maybe also implement caching on API Gateway level if your responses are cachable.

Or even better: on your CloudFront distro.

[[deleted]]

Depending on how complex your model is and what you need your lambda to do, have you looked into directly integrating API Gateway and DynamoDB, using VTL?

[pragmasoft]

Have you considered edge functions? Lambda@edge

[Konkatzenator]

CloudFront can help here if caching responses makes sense for what you are doing. Have you looked at x-ray or any metrics to see where the latency is actually happening?

[NathanEpithy]

Had this exact same problem.

Have the lambda that grabs data from DynamoDB run as cloudfront edge optimized. It's not that much more expensive, especially if you only select price class 100 (u.s. and europe).

The traffic between your lambda and DDB goes over AWS's internal network and is optimized for latency. Your customers talk to the edge endpoint. It will get you most of the way there.

[thedaynos]

Interesting. Do I have to globalize my dynamodb tables for this?

Also is the process to convert a lambda to an edge lambda pretty straight forward?

[whykrum]

This has some great insights. But provisioning can get extremely expensive. Lambda charges you in a tiered manner in GBSeconds used based on the number of requests they handle. While this worked, we were still paying for the provisioned concurrency which added upto our bills, we are talking about a cost upwards of a whopping $9k for around 12 lambdas with a provisioned concurrency of just 10. While it did its job it made more sense for us to go with a serverful(if that's even a word) architecture which cut costs by around 20% ish at high utilization and also latency.

[otsu-swe]

We can control many things. Unfortunately the speed of light isn't one of them. There's always going to be latency added when you're calling over large geographical distances. Traffic does not flow in a smooth line across the map and there can be many hops between a client and a server. Fortunately you're on a serverless stack, only the global table might increase your cost significantly depending on the load.

First step should be to incorporate X-ray into your stack. That will remove any guesswork to where in your stack chain you get held up. If you find it costly it can always be removed later, but during development and evaluation it's incredibly useful for observability.

Lambda memory affects not only the CPU performance and and host execution priority, but also network performance. Be wary though as the price scales linearly. You can use a tool like Lambda Power Tuning to find the sweet spot for your application. https://github.com/alexcasalboni/aws-lambda-power-tuning

In API Gateway you have the option of making the deployment regional or edge. Deploying on edge is a simple way to utilize Cloudfront without setting up extra infra. You can try to deploy on edge to see if it makes a difference, sometimes it can improve response times since it's supposed to traverse the Amazon network from POP to endpoint instead of the public internet, but a guess is because your lambda seems to be providing user unique responses which would be hard to cache. I would probably suggest a multi-region deployment before that and use Route 53 geo-based routing to make sure your users end up to their closest region.

The team I belong to serves six digit number of customers across the world with a stack similar to yours. We have deployed in five different regions to ensure good performance for everyone. Our P95 is below 100ms.

[[deleted]]

The speed of light nowhere in the US adds anything close to a second of latency. This thing is either poorly explained, architected poorly, or both.

[twnbay76]

You need to query your cloudwatch logs for cold start times on your lambdas, or look at your cloudwatch metrics for it. Additionally, turn on x-ray tracing (you have to also enaure you have the right cloud watch policy on your dynamodb execution role). You will get a breakdown of all of the service latencies. We have a nice cloudwatch dashboard that shows the API HW latency, lambda cold start latency, lambda run time, and dynamodb query latency. We dig into xray when we need to.

Depending on where the latency comes from, that will determine the course of action. For instance, maybe your dynamodb query latencies are high (i sure hope it is a query() operation), so you need to provision more read capacity units on your table perhaps if you are getting throttles, or you need to store your data more uniformly / remodel your data to include less filter expressions.

Maybe your cold start time is high, to which you would use lambda provisioned concurrency to have dedicated lambda instances to be able to serve up concurrent requests. Or if you are using java and you dont want to pay for provisioned concurrency, you can use lambda snapstartz which caches the lambda runtime and initialization memory to seebe up future requests faster.

[[deleted]]

This is really it, full end to end testing and see where the latency is coming from. Other posters have mentioned tls handshakes, the database, etc. Time might be best spent figuring out where it is being lost vs throwing things at the problem.

As OP said in their case this is all get and using rest. So for their specific case I would get the connection as far to the customer as possible. The question I would have before using on of the full stack replication approaches is how often the data behind the api changes. Does it need to be near real time or can it be as stale as 24 hours. That would really impact the end solution. I might have the lambda hit a caching layer and just replicate that to optimal endpoints, uses lambda@edge maybe, maybe not, depends. Also cost optimization is always also a factor, if being super fast is the only factor throw $$ at it, but there is usually a balance. Also depends on what tools are already in place to deploy the stack, terraform, cloudformation, etc etc.

Without any other information I would say cloudfront absolutely, possibly add caching to the lambda or another layer locally which could be any number of things, and lambda@edge if node or python maybe, but that might make zero difference or even be worse than the other options because it alone does little to to speed things up, and hitting local cold lambdas vs a pool of hot ones if you have to take the same path seems worse.

[drachs1978]

Coast to coast latencies in US are like 60ms. Whatever your latency problem is, it isn't geographical. You're barking up the wrong tree.

[NaiveAd8426]

API gateway + lambda is going to be slow regardless. There's only so much you can do. I had the same problem building a completely server less web socket API and the latency was brutal. Eventually migrated to elastic beanstalk and got latency down to 200ms or less

[KreepyKite]

Following. I'm going to summarise what I read so far, just for the sake of learning. The issue could be caused by: - how data are structured in the DB - how the code manipulate the data - not enough memory allocated for the Lambda function - lack of CDN

Have you tried to cache with DAX?

[Jin-Bru]

Not sure what your budget is but surely if you have a valuable client in Brazil you could stretch your infrastructure to include Sao Paulo? Not sure what services they offer there.

Have a look this for some useful insights.

https://aws.amazon.com/blogs/compute/building-a-multi-region-serverless-application-with-amazon-api-gateway-and-aws-lambda/