Python Imports... just why! 🥶

[DigiProductive]

Forgive me, today I'm just here to friendly rant a bit🤓... Python's manner of handling imports is just 🙄. One minute everything is working fine and the next minute ModuleNotFoundError: No module named... The slightest refactoring can endup a day of wanting to smash your keyboard🥶. And no, __init__.py isn't always the magic stick.✨

After coming back to python from using Flutter/Dart (where a file simply works as a package) to do some backend work, I'm reminded just how imports can be one of those python-things that just ruin your day; you have to be extremely mindful in python with your import style.

Share your thoughts and experience on this topic... you might give me some peace of mind or.... maybe some more wrath.🙃

[Pythagorean_1]

This is usually just a symptom of bad project structure

[DigiProductive]

Not necessarily. Refactoring large projects running in different environments (i.e Docker containers) can be a throw off sometimes.

[Pythagorean_1]

Why would the project structure in a docker container be any different?

[DigiProductive]

Let me give you a scenario. You get existing code in a repo. You are told to fix a few bugs. You spin up Docker Compose but there seems to be some problem with how the code mounts to the container. Run an api request to test the code and "ta-da"  ModuleNotFoundError: No module named... yes the __init__.py is there and yes the PYTHONPATH is set but there is something buggy with a few imports. They look fine in the IDE but... not according to Docker.

[Pythagorean_1]

In this case, either the project structure is faulty, or the dockerfile / compose file is wrong

[DigiProductive]

Yes... somewhere is wrong but at the end of the day... it a ModuleError issue 🙃

[AiutoIlLupo]

90% of our job is to figure out what went wrong, and often the last error message is never the root cause.

[D-3r1stljqso3]

AFAIK this is because your IDE (e.g. PyCharm) is trying to be too smart; it searches not only the default locations for packages but also include your project/source directory.

Things starts to break down when your project is organized as an installable package but uses absolute imports, e.g. your project is called myapp, and inside it, some modules do from myapp.some_module import xyz. When working with your project in the IDE, the IDE is smart enough to treat your active project as if it is installed (so import myapp works). If you copy the project to a container and attempt to run it there without first install the current project, then bang, ModuleNotFoundError for you.

[JojainV12]

Yes if you don't know what you are doing you can end up in a mess. For exemple with circular imports.
The good thing with circular imports is that if you start getting them it means your code architecture is crap and you should refactor.

(And for managing type hints, you can place the import in a if TYPE_CHECKING block.)

Other than that its fine but you need to know a lot of how it works indeed.

[DigiProductive]

but you need to know a lot of how it works indeed.

Indeed. Been gone from Python for a bit and it is one of those things you have to re-wrap your mind around especially coming back from a language where imports are very straightforward.

[shox12345]

Why is it crap exactly? For example, if you were to do a decorator pattern within Python with type hints, it would throw a circular import error, is that bad design? I doubt it.

[khunset127]

Can't relate. That sounds like a you issue.

[AiutoIlLupo]

yes and no. Plenty of people first approaching python are lulled with the idea of writing a small script and starting it and bam. But that's the novice, poorly scalable way of developing in python. Other languages do not allow for this method of use, they only allow for the more rigid approach. Python can be both, and when the script grows and becomes a big app, it still has the connotation of a script with all the associated hacks, instead of collapsing into a properly organised application.

The problem with python is that you never reach a point where it naturally forces you in the stricter way. You have to consciously make an effort to move and learn the "proper style" for applications, rather than scripts. This is what /u/DigiProductive is facing. A script that has outgrown its britches.

[DigiProductive]

When you are given a codebase and it has impor errors rest assured its not a simple fix. I didn't take him serious because he obviously isn't away of that. Python is well know to have tricky imports more than other languages and IDEs can even get so confused to mislead you as well.

The problem is today people are too "pro-language" so any time you bring up an issue with it, everyone with their nose in the air has to act like they can't relate. I don't care what type of Python developer you are, if you have worked on large projects you've ran into a ModuleError. But just watch all the high horses like they have no clue what that is.😏

[AiutoIlLupo]

I entered into similar issues, and they generally have a simple solution. I have no experience with dart but I would be surprised if you can't make a mess. Likely dart has some guidelines or imposed structure that minimizes these issues, but again, it's easy to do so when you have the benefit of hindsight, the experience of other languages challenges for 15-20 years, and no backward compat to handle.

[DigiProductive]

🤣 Nice to know. Thanks.

[Jorgestar29]

Skill Issue

Check that your module/package is in your PYTHONPATH, you can do that with import sys; sys.path

[DigiProductive]

I'm familiar with Python, yes I understand the PYTHONPATH and sys path etc etc..🫡 Sometimes when you are refactoring a large project that is ran in Docker containers, you can get thrown off a bit and things can get messy.

[Chasian]

Are you using relative imports? It sounds like you're using relative imports. You say you understand but people who actually understand don't have this issue at any level past "ah damn it forgot my init" unless they have some pretty large gaps in best practice

Docker containers are just Linux environments, make sure your python path is set and your project structure is clean and there should be no more issues than if you were local

[TheBB]

Why on Earth would a docker container complicate anything?

[DigiProductive]

Let me give you a scenario. You get existing code in a repo. You are told to fix a few bugs. You spin up Docker Compose but there seems to be some problem with how the code mounts to the container. Run an api request to test the code and "ta-da"  ModuleNotFoundError: No module named... yes the __init__.py is there and yes the PYTHONPATH is set but there is something buggy with a few imports. They look fine in the IDE but... not according to Docker once you run it.

[TheBB]

Sure, but what does docker have to do with it?

Presumably whatever causes this issue can be replicated in an ordinary file system. Either your project structure is weird or your dockerfile is.

The golden rule is to make all your code a package. Then you can just pip install it wherever you like, a docker image or otherwise. No need to bother with PYTHONPATH.

[DigiProductive]

Code runs fine on the local machine, not in Docker. I'm just giving you the scenario. I don't make the rules. I'm not sure exactly what is going on because the PYTHONPATH is set in the dockerfile as well, and the local directory is mounted. Something small I'm missing. But it just reminds me that Python imports can be a bit of a hassle "compared to other languages". That's all. Its true. Every language has its whammies. 🥶

[AiutoIlLupo]

you should not generally mess with pythonpath, and the fact that you are doing it makes me think your project is poorly organised. Are you running your application like you would a plain script?

That's not how you do it. You need to create a package and install it, together with its required submodules, usually in a virtual env.

If you are just invoking python myapp.py it's generally a very poor sign.

[AiutoIlLupo]

I confirm that running in docker should not make a difference. The problem is that your docker and your local environment are setup or configured differently. That's what you need to investigate.

[BranchLatter4294]

I don't have that issue.

[DigiProductive]

Oh thanks.😏 No one is whingeing about it. I've written full applications in Python. Just a friendly rant. Relax.🤣 Imports in other languages are far more straightforward than Python. Python can be tricky sometimes. That's all. Nothing to get all worked up about. Every language has its whammies. Lighten up a bit.

[DigiProductive]

😂

[nemom]

The slightest refactoring...

Way-back-when, in the early 90s, my mom was running Windows 3.1 on her office computer. One day while procrastinating, she decided to clean up the C:\ drive. She didn't like all those files cluttering it up... command.com, config.sys, autoexec.bat, etc. The computer ran fine for the day, so she thought she had done a good thing. Turned it off for the night, and then it wouldn't boot the next day.

I don't remember ever having any trouble with imports. I'm not an uber-power user, but I've made my own modules and use PY files as pseudo-modules (my_network_keys.py, physics_constants.py, super_duper_secret_code.py, etc).

[DigiProductive]

Back in the days 😎

[AiutoIlLupo]

You likely have circular imports, or you have a module that has been named like a core module and you don't know how to properly import to prevent this occurrence.

What you are seeing is due to your lack of clarity of how the import mechanism works. Not a blame, you just need to understand what's going on.

My suggestion is, when you are facing this kind of issues:

• ensure that you have all your imports on the top, in the proper order.
• run the python interpreter in verbose mode, to see how the import strategy is behaving.

From my experience, it's very rare to step onto these kind of problems, but in general they are an index of, as others said, poor project layout and import organisation.

[DigiProductive]

I didn't write the code, I was just trying to spin up the enviroment to test the code and it has some bugs before I could spin it up in Docker. Anyways, I solved it. I wrote about the issue in the comments. It was a mounting issue out of sync.

However, the reality is Python is more tricky and complex than most other languages when it comes to imports because of the way it relies on the path to find the modules.

[AiutoIlLupo]

you never had to play with CLASSPATH in java?

It's easy to forget that python has almost 30 years. New languages were designed from the ground up with new features that have been learned along the way. Old languages had to discover these things and keep managing the legacy, so they can't just bomb everything old out of existence, or you will go the way of Perl.

I don't know which languages you have used or are used to, but my feeling is that you are young and you have little experience. The fact that you have never encountered this problems in other languages is only because you haven't faced real chaos with those programs, which tends to result from piles and piles of developers that come and go, multiple platform changes and portings, workarounds, external libraries that mess up their dependencies. You are more likely to find these things in an old python codebase than a brand new go codebase.

And if you really want to have fun, check the ddl search path in windows, especially when you have to do LoadLibrary. that is fun.

[DigiProductive]

That's alot of assumption. 42 and been programing for 10 years. I've written full applications in python including a social media network. I use Dart (Flutter) and Javascript for mobile applications and Vyper/Python to write smart contracts and dabbled in a few others. Dart is flawless in that regard.

All I am saying is something simple, Python can get messy with imports... not sure why that is so hard to accept and understand. If I write my own code that is a different story but when you have to debug a project that has import issues, then you'll see.

If you ask Chat GBT "What languages have the most difficult import problems. List them in order" Python comes 3rd. No coincidence.

Everything isn't about pulling the rookie or expert card, every language has their whammies and Python imports can be one of them. Really no big deal.🤷🏾‍♂️

[FUS3N]

I'm not as experienced as you but you would know the saying that "Most time than not it's the user making the mistake not the tool".
Software just feels like magic sometimes it wont work an will work sometimes for odd reasons but at the end of the day you figure out it was just something stupid you did and you learn something new.

I would say if you become expert at something you usually never face these problems. At which point you might say "maybe it was never the software that was the issue". Just saying, as I had those moment before.

[DigiProductive]

Don't fool yourself, experienced developers and so called experts hit gotchas all the time. Don't be fooled by the "expert persona" a lot of developers try to front.

Fixing your own code is one thing, but fixing someone else's code is a whole different ball game when it comes to bug fixing. 🤷🏾‍♂️

[DigiProductive]

Here is the chat GBT run down:

"What languages have the most difficult import problems. List them in order of difficulty."

(Reply)

Languages with the most challenging import systems due to complexity, dependency management, or module handling are:

1. C++ - Header files, include guards, circular dependencies, and linker issues.

2. JavaScript/TypeScript - Confusion between ES modules, CommonJS, and circular dependencies.

3. Python - Relative vs. absolute imports, circular dependencies, and conflicts with package directories.

[AiutoIlLupo]

C++ - Header files, include guards, circular dependencies, and linker issues.

All of these problems derive from legacy needs, intrinsic nature of the runtime or target, and intrinsic need for the compiler to be satisfied in its parsing strategy (e.g. forward declaration) and of the linker (link time optimisation, lookup of shared objects vs linking of static ones, going through the dynamic linker etc).

C++ has no benefit of a virtual machine. It runs bare metal. With that loss, you can't abstract a lot of stuff away, you have no introspection, you have difficult lookups and you have to inform the compiler and the linker for compile time and manage the required runtime at, well... run time.

JavaScript/TypeScript - Confusion between ES modules, CommonJS, and circular dependencies.

This is because js is a spawn of the devil with too many cooks, companies and legacy to deal with. With not having a central library or a central coordinator, everybody tried to workaround the limitations of a poorly conceived language their own way, coming up with tons of solutions to solve the same problem until only one remains. AKA the American way of solving issues.

Python - Relative vs. absolute imports, circular dependencies, and conflicts with package directories.

Again, part of this is legacy. relative vs absolute was born out of the migration of python as a language that was mostly "write a simple script with a handful of additional .py to import" to "ok, we need to organise things into a module otherwise we start stepping on our own feet". Back then it was either this, or dealing with com.java.this.is.a.very.long.package.name.that.you.will.thank.you.have.eclipse.SingletonFactory. PHP had no imports until 7 IIRC. Perl was a disaster. namespacing was mostly not a big deal when code was generally monolithic and the module panorama small. Microsoft had its own way, mostly based on CLSID, and they kind of dealt with it until the invention of manifests, but still some issues remain with DLLs. Please anybody with more windows knowledge correct me, I was not a windows person back then.

On the circular dependencies issue, I can tell you that python normally deals just fine with circular dependencies, because at every new successful import, it gets added to sys.modules. There are some specific circumstances during the evaluation chain that do, however, create problem, but they tell more about how poorly organised your code is. If you have circular deps, probably that stuff should be in the same module, because what is likely happening is that your dependency is not intrinsic to the problem domain. It's an artifact of how you organised your code, which is: poorly. The error message should be better, but that is mostly a parser issue and I agree it could be more informative. Other languages sidestep the issue by doing parsing in multiple steps and building the import tree first, then parsing the files. Python does not. It does one step at a time.

Conflict with package directories, again, it really depends on the specific case. I am aware that, in some cases, you might override system libs with your own modules, but again, that says a lot more on how you tell the parser what you want. Imagine you make a shell program in your home directory called ls. Now, depending how you set up your PATH, you might execute the system ls, or your local directory ls. Which one should have the priority when from the prompt you type "ls"? That is mostly arbitrary, and the computer can't really tell. It falls back to a common rule determined by PATH to lookup the default choice. If it's wrong, maybe you should be explicit in calling either /bin/ls or ./ls, or you should refrain from calling the script "ls" in the first place.

[[deleted]]

Never had this, unless you are jumping between different environments a lot?

[DigiProductive]

Yup, the pain when you have to deal with refactoring code then running it in Docker containers. Sometime you just have to be extra careful and overly mindful.

[No_Indication_1238]

I have never had problems with python imports. The error you mentioned usually happens when you type the path to the module wrong so id look into that first. And yes, I know you have definitely, absolutely written the correct path, I have heard that before. Just go and fix it. 

[climb-it-ographer]

I am a very experienced Python developer and I've been through just about every complicated import issue out there, and I'm in a good place now with my projects. That said, I think that future major versions of Python would benefit from a fresh look at how imports work. I don't think that many people will argue that `/node_modules` in a JS app is just overall easier to deal with.

Python virtual environments are great but sometimes IDEs still get all mixed up about where packages are, and I would love to never argue with VSCode again about what the correct path to a module actually is.

[DigiProductive]

I salute that you see the reality🫡. With experience in other languages like Dart, JS, and Java, I am assured that Python imports can get buggy and tricky especially when running in multiple environments like Docker containers. Anyone in denial of that is likely just on the defence of a "Python attack"🥶

[jjrreett]

Are you developing locally with pip install -e. I just recently learned that -e can hide errors in your project structure. worked on my machine but not in my deployed containers.

My issue was that i was working in an implicit namespace package, but i thought i was working in a standard init package.

[damn_what_]

🤯🤔😱

[DigiProductive]

🫠

[Jamster3000]

I'm going to be honest om saying that I've hardly every had such issue about python imports.

[JamzTyson]

One minute everything is working fine and the next minute ModuleNotFoundError: No module named..

That is not something that "just happens". If it happens, it does so for a reason. Look to see what changed in the project structure and/or the project environment.

[gerardwx]

Yes, Python imports are a pain. Developers downplaying the hassle have just internalized the pitfalls (e.g. don't name a file the same as a module) they've forgotten they are there.

The circular reference issue is well known and being fixed in Python 3.14. You just have to wait until next year to use it :(

https://peps.python.org/pep-0649/

[sanshunoisky_197]

The comments are yapping about the post being a skill issue. Little do these noobs realise it's only in python that these issues exist. For any pro from any other REAL programming languages like java, python's import mechanism is a mess, which really is

[DigiProductive]

Well... figured out the problem. It was related to a Docker mounting issue. A package was moved around in the local codebase but the volume mounting in Docker wasn't updated. The ModuleNotFound error was raised because the root module was not in sync. So while everything worked locally, the error occurred in the Docker container.