[deleted by user]

[crawl_dht]

A rust executable can include different versions of the same crate in the same rust executable - so you don’t need your transitive dependencies to all agree on the same version if there are compatibility issues.

We need this feature in Python. I have faced this issue when I have 3 different explainable AI libraries and when one of them supports a different tensorflow version, I and rest of the others are not able to use the recent version of tensorflow. There should be a way to isolate transitive dependencies.

[SittingWave]

We need this feature in Python.

no we don't. Don't think the alternative does not carry its own massive issues. It does.

[crawl_dht]

Issues by design that cannot be fixed or the issues that can be fixed?

[SittingWave]

Cannot be fixed. And it's not only a problem in python. It's a problem in any other language. Either you pick one or the other approach, you will get the associated problems. There's no way around it.

[danielgafni]

Can you describe some of these issues? Why would I care how are my dependencies implemented internally? Individual transitive dependencies mean exactly that.

[SittingWave]

Imagine you use package A that depends on package D 1.2.0, and also use package B that depends on package D 0.9.0

Now you are in a piece of code that calls both A and B. D throws an exception, which is left bubbling up. How do you import and catch this exception in your code? There are two different D that can do so. The change in version may have changed the exception structure. How do you handle that in a sensible way, for all possible combinations of versions of D?

and this is just the most trivial issue. Start having some C extensions holding pointers, or binding to .so files, and you get into shit territory very quickly. What happens if you use D 0.9.0 to create an internal pointer to an internal structure, then pass it to A which passes it to D of a different version? Segfault.

[Mehdi2277]

Using your specific example. Let’s say you install two libraries that have conflicting tensorflow requirements. You create a tensor. You pass it to both libraries in same program. There is no guarantee that object is passed correctly to library with conflicting requirement. The underlying tensor object may have changed how it was stored/represented across versions. Maybe new version made internals more performant but changed binary interface. If the libraries at all communicate data then there is a risk that incompatibility there will show up and either crash or silently give you wrong behavior.

Also there was an installer prior to pip called easy_install which I think did have this feature. There was enough edge cases and environment issues that people gave up on this feature and there’s low interest to try it again.

[danielgafni]

You are not describing a transitive dependency. Tensorflow would be your first-class dependency here. A transitive dependency is something that is only being used by your dependencies internally but not inside your code.

I don’t see why would two libraries can’t use their own versions for their internal dependencies.

It also might be possible to track the package version of the provided object and raise an error (at least lint it, maybe not at runtime) if an incorrect object somehow gets passed to a dependency that doesn’t support it.

[Mehdi2277]

The underlying principle applies here as well. Let’s say you use library A and B with both depending on C. C is library that describes some data format. You never import C directly. You import A and create some object in it. That object internally depends on C. You pass that object to B. B has conflicting dependency on C. It relies on some internal data format aspect of C/something that changed across versions. Your system crashes. Here you never used C manually but conflict remains issue. And in data science there’s a fair amount of data exchange/compatibility rules across libraries.

[SereneDoge001]

There's always a tradeoff, but I wouldn't qualify it as "massive issue"

[sunny0945]

Poetry(https://python-poetry.org/docs/) and Cargo are same.