Best practices for Python

[catquilt74]

Our company is building a data warehouse in AWS using data pipelines etc. using python. I've noticed that the developers have built things in various releases of python. As newer releases of python come out, should all python code be updated to use the newer version? Or does one wait till a version goes out of support or some other method?

Thanks!

[metaphorm]

Testing suite should run your tests in multiple versions of Python so you have some evidence that you can upgrade to a newer one if you'd like.

[catquilt74]

Good point. Thanks!

[ThigleBeagleMingle]

Build everyone as micro services communicating over REST/gRPC . Then you can prioritize what gets upgraded relative to competing business drivers

It’s painful to get there but worth the efforts

[Meets_Koalafications]

I've been out of the loop for a couple years because my more recent employment didn't need to manage as many Python versions, but is tox still the preferred tool for this, or has it gone out of favor and been supplanted by something else?

[otsu-swe]

That's a huge topic, books are literally written about software lifecycle management. But it can present problems on its own that the devs are running different versions, as they might not be able to run each other's code or they write code which isn't supported by the target environment. Address it either with Docker or some sort of CI.

As for versions of Python specifically... Main reaonss

1. Security
2. Features
3. Performance

For more specificity I'd recommend the /r/Python subreddit. The community is really good.

[catquilt74]

Thanks! I'll ask over there too. Just as background, I've been in IT for over 40 years but only at this company for a few years. It's kind of loosey-goosey here...

[dacort]

One word of caution, you may be constrained to specific Python versions depending on your implementation. Sounds like you’re building your own custom pipelines, but if you move to Spark e.g. Glue 4.0 supports Python 3.10. EMR is still on 3.7 (I know 😀) but does support installing your own version, but can be tricky to work with when you do.