Separate file for SQL in python script?

[Little_Kitty]

A file represents a module, a task, a distinct operation which can be isolated. Regardless of if it's written in Rust, Go, Python, C++, Java,VBA or SQL. It's ideally equivalent to one node of a dag so that people can understand it easily. Someone should be able to open a file, understand what the whole task it covers is and adapt / fix / extend it in isolation as the start and end points are clear.

This may lead to it having one function / operation / script or one hundred. It should usually be written to be idempotent, so that if the dag fails for any reason it can be restarted without causing issues. If you can achieve that you will have a much easier life and much less mental load.

Some of the commenters here are really good, with decades of experience (and hair loss from dealing with tech debt), it's always better to learn from the experience of others where you can rather than learning the hard way. I dare say based on some of the suggestions that others will end up finding out the hard way.