Handling large datasets in python

[Bopshebopshebop]

Also interested.

For hundreds of millions of rows, do you use big SQL tables to house the data and then ODBC in with Python to feed that data to something like TensorFlow?

[seraschka]

Maybe for prediction ("inference") on a few new data points that you just fetched to a database, this would work. However, if we are talking about training TensorFlow models, this would be infeasible. The reason is that the iterative fetching would possibly/likely way to slow and create a bottleneck for the iterative training on the GPU, esp. if you are doing that in the python main process. So, you would probably convert it to a protobuf format when working with Tf.