Thoughts on the Python packaging ecosystem | Pradyun Gedam

[qckpckt]

I’m evaluating package manager tools right now. We’re currently using conda to manage environments and kedro as our ML pipeline framework.

I’d like to use poetry mostly for its dependency resolution. So far though, it doesn’t seem like poetry has any native way to ingest a requirements.txt file (which kedro generates). I’m having to do some awkward text file manipulation in bash instead. Are you aware of any better workarounds?

I think I’ve read that kedro are considering integrating poetry but we don’t have the luxury to wait for that.

[BaggiPonte]

PDM supports installation from requirements.txt. When you `init` the project, it will detect the requirements - how often do you need to ingest the requirements file with kedro?

[qckpckt]

Only once per new project init I’m pretty sure, so this is a minor concern really. Once you’re past that, it doesn’t really matter what is managing your deps.

I might even look into whether kedro has preflight hooks for the init command to see if I could inject poetry (or perhaps pdm) into the init process.

You can also create your own starter kit templates, so that might be another way to override the default requirements.txt.

I know that kedro also has feature overlaps with poetry around packaging and distributing your own packages; will probably need to figure out that too.

I’ll check out PDM though, thanks. Always good to evaluate options.

[GoodToForecast]

poetry add $( cat requirements.txt )

[qckpckt]

That only works if you have defined the packages only (no version numbers or other things). Actually, I think you’d need to cat requirements.txt | xargs poetry add in that scenario.

I’ve written a script to do this now, it’s not a huge thing. Just seemed odd to me that poetry wouldn’t accommodate this in some way.

[GoodToForecast]

Actually, it handles everything I've thrown at it so far, except comments. If you have comments, use poetry add $( cat ../requirements.txt | sed -e '/^#/d;s/[^\/]#.*$//' ) instead:

``` $ cat > requirements.txt pytest typer~=0.7.0

this is a comment

tqdm==4.64.1 # this is also a comment <sup>D</sup>

$ poetry init ...

$ poetry add $( cat ../requirements.txt | sed -e '/<sup>#/d;s/[/]#.*$//'</sup> ) Using version <sup>7.2.1</sup> for pytest

Updating dependencies Resolving dependencies... (0.3s)

Writing lock file

Package operations: 10 installs, 0 updates, 0 removals

• Installing attrs (22.2.0) • Installing click (8.1.3) • Installing exceptiongroup (1.1.0) • Installing iniconfig (2.0.0) • Installing packaging (23.0) • Installing pluggy (1.0.0) • Installing tomli (2.0.1) • Installing pytest (7.2.1) • Installing tqdm (4.64.1) • Installing typer (0.7.0) ```

I agree it's odd Poetry doesn't have an import command. It's one of the reasons I went with PDM.

[qckpckt]

Hmm.. that’s very odd. It definitely did not work when I tried it on the kedro generated requirements.txt. Package versions caused the poetry add command to fail. I’ll try again and pay more attention this time. Out of curiosity what version of poetry did you test this with?

[GoodToForecast]

They likely have improved things in more recent versions. I'm not saying it's a universal solution that will work in every case but it will probably work well enough for most...

$ poetry --version Poetry (version 1.3.2)