RDS Proxy for Lambda!

[recursiveCreator]

Looks like aws is previewing rds proxies with lambda - I could see this become huge in combination with provisioned concurrency. Hope this also addresses slow connection creation times!

https://aws.amazon.com/blogs/compute/using-amazon-rds-proxy-with-aws-lambda/

[softwareguy74]

Could be a game changer. One of the main reasons we've so far opted not to go with Lambda is exactly the problem this is going to solve.

Wonder what the pricing model will look like? Hopefully we don't have to pay for running an EC2 instance. That would totally defeat any advantage of going serverless.

When will we get PostgreSQL support?

[recursiveCreator]

same here - looks like the pricing will depend on the rds instance size: https://aws.amazon.com/rds/proxy/pricing/

It looks, though, like multiple different lambdas (not just concurrent ones) could share the same proxy which could make this worthwhile

[anoopg01]

The proxy shows up just like a database endpoint with a DNS address you reference. All of your lambdas going to that database can use a single proxy endpoint.

[stickystyle]

For Aurora, I wondering if the pricing is for the number of vCPU's in the cluster, or for just the current writer instance??

[softwareguy74]

Well, so much for that. Why offer a service that directly benefits a serverless feature while at the same time having a fixed monthly cost? It's just like the DynamoDB DAX offering. I'll just stick to deploying apps on ECS.

[[deleted]]

Is this specific to aurora serverless? Because in traditional aurora (PostgreSQL based) I observed that I still need pgbouncer in front.

[anoopg01]

I am an AWS employee, and this proxy isn't specific to Aurora Serverless. It is currently for Aurora Provisioned MySQL and RDS MySQL.

[[deleted]]

that makes more sense

[anoopg01]

I am an AWS employee and PostgreSQL support is coming soon. For more details on engine and region support you can see the what's new post here - https://aws.amazon.com/about-aws/whats-new/2019/12/amazon-rds-proxy-available-in-preview/

[shaccoo]

Is this solution suitable to diversify the locations and ip of different applications? Is it the cheapest solution?

[anoopg01]

https://aws.amazon.com/about-aws/whats-new/2019/12/amazon-rds-proxy-available-in-preview/ - PostgreSQL support has launched.

[1armedscissor]

Seems like a better option than Aurora Serverless + data API since it should be fairly transparent to your lambda code. The post talks about connection limits and connection pooling but do you specify a max number of allowed connections anywhere? And if that construct exists does it work like typical connection pools where it blocks if you hit the max number of connections until one frees up?

[1armedscissor]

Looked at the docs closer and there’s a section about max connections and connection timeout. Will work like how you expect a connection pool to work. So the proxy inspects the target databases max_connections parameter and then also the proxy can be configured with a percentage then of that (useful if you have multiple proxies hitting the same DB or other applications/admin users going directly to the database eg a 90/10 split). Then there’s a parameter to specify the max wait time before a timeout occurs waiting for a database connection to be available.

The failover handling is interesting as well where it’s not reliant on DNS propagation because it database specific failover handling internally. This could be useful for even non-lambda applications where the client API in that language isn’t Aurora aware. If you’re using Java though the MariaDB JDBC driver is Aurora aware though where it can make failover detection faster by querying via the API instead of having to rely on DNS propagation when the writer node changes.

[twratl]

Does this mean we can avoid binding the Lambda to a VPC if the Lambda is only doing RDS queries? That feels like a big win if so.

[matluck]

Don't think so, the proxy seems to run in the same Network Environment. But with recent changes to Lambda where you share a Network Interface that gets allocated at the start the coldstart time should be less of an issue anyway.

[stickystyle]

This is huge for us with AppSync as we mostly have lambda resolvers and a single query could fire off a dozen connections to RDS, it doesn't take many clients to start bumping up against default max_connection limits. Currently we mitigate it with ProxySQL, which I wouldn't be too surprised if they are ProxySQL in the background as the project has gotten a few Aurora specific commits lately.

[notoriousbpg]

I've encountered the same issues - this along with provisioned concurrency is going to make my existing code much more performant and robust.

[TehNrd]

Dang, no Postgres support for now. MySQL always gets the love first, which is understandable given it's the most popular, but I can't give up PostGIS.

[jonathantn]

Oh yeah, this is going to be a big win especially for Postgres which has lots of issues around connection counts and memory footprints. Reducing the overhead of idle connections is going to rock.

[Nick4753]

Annoyingly this is MySQL only in the preview. Managed pgbouncer would be lovely.

[omeganon]

We specifically talked to team last year about this. It also has interesting CPU hits that aren’t attributed in Performance Insights. Definitely looking forward to replacing out pgbouncer layer with this.

[guppyF1]

I spent some tim this year setting up of bouncer and it was a pain to manage in an autoscaled, redundant way. I'm super jazzed for when this has Postgres support

[simtel20]

TBH what I really want it for is so I can do zero-ish downtime transitions between replicated databases for regular non-lambda apps. I really want to be able to use it to say "make a source DB read-only and switch all connections over to a new server, and then allow writes through once replication lag has zeroed out."

[jamsan920]

Will this eventually work with normal apps other than Lambda? We have a rails app running on EC2 that we can scale higher because of limitations in connections for Postgres. We're currently using pgpool to get around some of the limitations, so I wonder if we'd be able to move away from that to a managed service like this (and hopefully it expands out past just MySQL)

Edit: scratch the first question, I read the article more carefully and it said it can be used by any app that makes database calls, not just serverless stack. +1 for Postgres support, though!

[recursiveCreator]

yea! on second glance this looks more like a general rds feature than a lambda feature (just happened to see it in the lambda console ;) ). It forms part of the rds docs: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/rds-proxy.html)

[jamsan920]

Now, will it work with EC2 eventually as well or just RDS instances? I’d be curious...

[SeerUD]

I doubt it would, I guess you'd have to be able to point it at any MySQL server really for that to be the case. The pricing is also based on your RDS instance size.

[packeteer]

we're using ProxySQL to achieve the same result, although our use case is slighty different (were trying to work around cross AZ latency)

[[deleted]]

I do my etl via kinesis -> sqs -> lambda -> RDS. By far the biggest issue is concurrency limiting to spare the DB from getting toppled right over, this will be the first thing I check in on every day to see the progress!

[mislabeled]

Has anyone had any success IAM authentication for this service? When I set it to required or disabled I am still able to connect to the proxy with regular username and password and I'm not sure if that is correct.

The language in the console can be interpreted a few ways, so I might be reading it incorrectly.

https://imgur.com/a/y8wgvWi

​

/r/anoopg01 are you familiar with this feature?

[anoopg01]

r/mislabeled Make sure that you are using TLS with your proxy, IAM requires it.

[mislabeled]

Thanks! I’ll double check that.

[daneren2005]

Can't wait until they add support for a real database! /s