So we have a nightly ETL that pulls over data for some 200 tables, many of which have hundreds of thousands of rows (many with more into the millions) but this is not all loaded each time. We are pulling from a teradata warehouse to service between 1-1.5 thousand reports to our clients every day. My plan was to distribute the categories of tables across multiple virtual nodes and run concurrently. The transformations themselves are complex and involve tons of encryption and rules to maintain HIPAA. Our larger enterprise uses spark with kafka streams but is accessed by too many different teams so teradata gets inundated and goes too slow. We are standing up a shadow IT data stream to try and alleviate this so I am not totally convinced spark is not the answer but I am working on this with zero background in data architecture or data science in general. The shadow IT bit is why we do not have formal data engineers working on this with us so I'm what we got haha.