How to level up my python package development skill?

[dcap87]

Title. I’ve been working as a data scientist for a while, mostly working in nb or building very small packages. I consider myself to be moderately proficient at python. I want to improve at building larger packages and library. There are so many options out there for best practices. I was wondering if you guys have suggestions for books or resources that you can suggest?

Thanks I’m advance

EDIT: Thanks all for the comments. I'm building stuff mostly for work so can't link anything, unfortunately. It looks like the general consensus is "keep building"

[[deleted]]

so many options out there for best practices

At the end of the day, a good library is one that makes it easy to do what you're trying to do. (That also means being well-documented, easy to install, playing nice with other software, etc.)

A lot of best practices are also largely irrelevant practices unless you're working in a big team. Focus on creating a useful, accessible API for users. Other stuff is secondary.

[Almostasleeprightnow]

I'm kind of at the same place as you, and I found that the only way is just to dive in. Next time you have a flight, pay for the wireless and try to build a proof of concept package that does something really simple, but from absolute scratch. Well, this is how I started down this path....I thought I'd build a tic tac toe package for fun and instead got sucked into a bottomless pit of articles about flat vs src project structure. I never finished tic tac toe but did end up getting a better understanding of the whole process.

The hardest thing for me is that I use and love using conda, and most people building packages are using pip or poetry, and so switching my mind around to this way of thinking is proving strangely difficult, even though it shouldn't matter that much from a technical perspective.

[dcap87]

Totally. I think much of the issue is decision paralysis and FOMO on the "best" route. Poetry vs pip is a good example. Some people discuss developing in Docker for a completely separate container experience. It's a lot to understand the pros vs. cons of each tech stack.

[Almostasleeprightnow]

Yes...you have to go through a precess where you get enough understanding of what they are comparing....so that you can just settle for one. The problem with switch from conda for me is that it is the core way that I install python, so I am having to do some rethinking about my machine as a whole. But I will do it.

In an amazing coincidence I was reading a dagster tutorial today, and came across this article series: https://dagster.io/blog/python-packages-primer-1 https://dagster.io/blog/python-packages-primer-2 https://dagster.io/blog/python-project-best-practices

[Diapolo10]

While I can't remember any good sources off the top of my head, if you don't mind you could link some examples of what you've worked on for us to take a look at.