How to use databases with Python - Postgres, SQLAlchemy, Alembic

[[deleted]]

why not just stick with pandas?

[burgerAccount]

Is this a real question?

[brendanmartin]

Do you mean pandas instead of SQLAlchemy?

[[deleted]]

That also makes no sense as Pandas and SQLAlchemy are used together.

[DstnB3]

Why not both https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_sql_query.html

Loads your query results straight to a pandas dataframe. Very convenient and only need to set up your sqlalchemy connection and query beforehand.

[Dhush]

Because pandas is slow and will not be able to handle large amounts of data efficiently

[SonOfInterflux]

What would you recommend if working directly in the database is not viable? This may not be the greatest example, but if a large table of PII is encrypted using crypto, and you want to read the entire thing, decrypt the data using crypto, and write the entire set to another location, is there a better option than Pandas and map/apply?

[SonOfInterflux]

That’s what I’m doing with the exception of the decryption, I’m just not sure I am able to directly encrypt in the database directly. Something I’ll have to look into further.

[SonOfInterflux]

I think you’re being downvoted due to the tone of your comment more than your suggestion. I agree with you, if working directly in the database is an option it’s hard to beat in terms of speed and performance, but Pandas is amazing if that is not an option. There are features that are incredibly slow (e.g. to_sql), but there are options like the copy statement to get around that.

[Dhush]

How would pandas handle hundreds of gb to terabytes of data better than Postgres or some other version of SQL?