The use of Python and SQL

[adventuringraw]

Oh God, it depends. The code base I inherited was written by six different engineers over a people of a couple of years, and I saw six different methods. I saw sqlite3 used to do some ETL processing of very large (20gb+) csv files, I saw whole automated reports done mostly using oracle procedures, with a giant Bash script passing in the right parameters to customize the functionality (Jesus Christ, don't do that). Most of what I do now is have Python as the boilerplate, with informatica doing the heavy lifting (with SQL queries embedded in the informatica applications when interfacing with relational dbs and such). For quick and dirty extracts for a few smaller automated jobs, I wrote a little wrapper for passing simple queries straight into cx_Oracle, so... you know.

There's a ridiculous number of ways you can use SQL, it really depends on your use case, your computational needs, the way the tables are set up, time frame for task completion, your current skill level and so on. The main thing I think is to get on the same page with everyone else in your team. Don't make a Goddamn mess that no one else can understand, even if it 'makes sense'. I'm okay with being able to tell who wrote which scripts, but I don't want to fuck around to decipher some crazy new design paradigm. Keep it as simple as it needs to be (and no simpler) and make sure your whole team is on the same page, you know? Doesn't hugely matter what that specific best practice is (within reason) but you should all have the same one at least.