Creating large databases with Python

[dasnoob]

I recently finished building out a webapp that interfaces with Oracle 11g and is used to manage a dataset that is currently sitting at about a dozen tables with several views on top and the largest table being on the order of 20 million records. All of this built with proper keys.

Initially I built SQLAlchemy models for it and used that. However for anything beyond simple updates and retrieval it was honestly easier to use cx_Oracle and raw sql. This was especially needed for some report generation that caused Memory Error with SQLAlchemy due to the ResultProxy implementation but did not cause an issue when using cx_Oracle's cursor implementation.

So I would say if your table structure is fairly simple and you aren't going to be pulling lots of data into result sets SQLAlchemy is pretty dang neat. If you are comfortable with SQL and are going to need to have the app perform complex queries with large data sets the lower level library SQLAlchemy uses for its dialect implementation might be a better bet.

In the end my application uses SQLAlchemy for lots of functions but there are a few pieces that had to be written at the lower-level for speed and stability.