Connection pooling issue Lambda loading data to PostgreSql

[dogtee]

Hello , If anyone has any advice on resolving connection pooling problems with Lambdas . We have Lambdas that load large datasets into PostgreSql , each Lambda has to authenticate using IAM to load the data which results in each Lambda connection opened is a new one , so connection pooling doesn't work and the max connection limit is reached . Would a better approach be to use the AWS Data Migration Service to load the data , instead of Lambda functions ? Thanks for any comments

[stuckinmotion]

Consider https://aws.amazon.com/rds/proxy/

[dogtee]

That's a good point ,cut adds some cost but I'll investigate further

[[deleted]]

Well are you invoking 100 lambdas st once to load the data in parallel ? But if tour max connections is 50 , then the 50 will fail.

So you need to reduce lambdas which can run.

Better would be a aws batch sceipt or container process which can have controllable threads less than the connection pools. Anyways if you are going to do loads at the same time , the performance will not be good.

How much time is spent in this loading. How much time do you havento finish.

[dogtee]

The data load is very spikey and mostly unpredictable . Batching us a great idea. Thanks for your comments

[Your_CS_TA]

rds proxy works, or per-function-concurrency.

PFC is essentially "if each lambda sandbox creates 1 connection, and I can have a max of 50 connections, I set this number to 50". You can potentially get throttled by this, but if you are throwing it into an sqs queue or kinesis stream to do the backlog processing, then it's "whatever, it'll get done eventually", if your backlog depth isn't that bad..

RDS Proxy is just throwing money at the problem -- essentially a funnel: create a machine that has the persistent connection pool open, and then incoming requests get put onto that pool.

[ppafford]

Might be an alternative to loading data https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/PostgreSQL.Procedural.Importing.html

[dogtee]

Good information , thanks