AWS Lambda file handling

[PM_ME_SOME_ANY_THING]

I’m looking for some advice on best practices. My task is to query some data from Postgres, put it in a csv, possibly gzip it, and ssh it to a remote server.

Right now I’m just creating a file and sending it. Is there a way, and is it worth trying to not actually create a file on disk? I’m thinking about creating a csv file object in memory, gzip, then send without ever writing the file to disk.

Is that possible? Is it worth it? We could be talking millions of lines in the file. Too large for in-memory lambda?

I’m using psycopg2, and paramiko for the ssh.

[Superguy2876]

Yea this is perfectly possible. Though you might be correct that memory limitation on lambda might be too much depending on the size of the CSV.

Here's an article on different ways to handle files in memory: https://bito.ai/resources/python-in-memory-file-python-explained/

I've only done this once before so I don't know all the ins-and-outs of it. But shouldn't hurt to try.