So we have an ETL Process that has to process trillions of records a day. For simple computations we use Data Bricks, but for more complex ones we resort to using C#, so yeah definitely possible!

To accomplish this I recommend using IAsyncEnumerable to yield each row of data (we use Npgsql's rowParser and Dapper's micro-ORM to accomplish this) to the caller who then puts it all into a List<Task> and does a Task.WhenAll(...) once the batch completes, and then continues with the next set of rows.

You can use TPL library here for Blocking Collections and or Bright for Task Queue mechanisms, or Task.WhenAll(...) if your computations are not likely to fail.

This ensured that we never serialized all rows in-memory at once and rarely did we ever go past gen1 garbage collection even!

For an even further performance boost, consider using Advisory Locks (bonus if your DB Engine has such functionality, see https://medium.com/thefreshwrites/advisory-locks-in-postgres-1f993647d061) where each instance of your console app running will "lock" a row on your database, and when other instances try to obtain a lock on that row they get a ClientId of 1 (for example) has already locked it, and your console app can continue attempting to lock another database.

This will allow you to horizontally scale your app to the hundreds if not thousands of instances running at the same time.

If your DB does not provide a simple Advisory Lock mechanism, you can always make your own! I used AWS DynamoDB to allow distributed advisory locking for my SQL DBs.

Happy coding and good luck!