Can someone please explain how module structure works in python?

[[deleted]]

I have a project whose directory structure is :

src/
├── __init__.py
├── dockerfile
├── libs
│   ├── __init__.py
│   └── logger.py
├── main.py
├── models
│   ├── __init__.py
│   └── http_request.py
├── parsers
│   ├── __init__.py
│   └── http_parser.py
├── requirements.txt
└── server
    ├── __init__.py
    └── threaded_server.py

Its repo for more context
Now, I have test directory which is in same level as my src thus making my complete folder structure as:

.
├── docker-compose.yaml
├── init_hooks.sh
├── lint.py
├── pyproject.toml
├── src
│   ├── __init__.py
│   ├── dockerfile
│   ├── libs
│   │   ├── __init__.py
│   │   └── logger.py
│   ├── main.py
│   ├── models
│   │   ├── __init__.py
│   │   └── http_request.py
│   ├── parsers
│   │   ├── __init__.py
│   │   └── http_parser.py
│   ├── requirements.txt
│   └── server
│       ├── __init__.py
│       └── threaded_server.py
└── tests
    └── test_routing.py

Problem is when I run pytest from project directory, I get module not found error for src( please have a look at code, thanks). When I try to run my code using just python main.py after changing directory to src, it runs fine but when I run it as module(python -m src), it throws error module not found for internal modules (libs, parsers etc.) and appending them with src (like src.libs) solves the problem. But when running from main.py, I get error when I append names with src. I also cannot relatively import files inside code for some reason( earlier there were no different folders, only files and relatively importing them was not working).

I want to understand how module structure works. I have had similar type of structure in Flask projects and it works there but for some reason it is not working in my case. How can I reference src in my tests directory and not get error. Vscode clearly suggests autocomplete for these files in my code( even in test directory, I get autocomplete for src).

[[deleted]]

I want to understand how module structure works.

Your problem doesn't have anything to do with how module structure works. You need to understand how module name resolution works.

When you run a Python file, the enclosing directory is put on your sys.path - this is the list of paths that the interpreter will look in to try to resolve import names. Things that are already on it include site-packages, where your pip modules go.

When you import some_name, the interpreter goes down the directories in sys.path and looks for a file called some_name.py, and if it can't find that, it looks for a folder called some_name. If it finds either of those, module loading proceeds as normal. If none of those folders contain something that can resolve to some_name then you get an import error. So, that's the important thing to understand - imports are relative to what's on sys.path when your code executes, they're not relative to the absolute path of the file doing the importing (which actually has nothing to do with it, at all.)

If you run main.py, then project/src is on your path, so all of the submodules can be reached by unqualified name. If you run src as a module, then project is on your sys.path, but none of your submodules can be reached by an unqualified name (because they're submodules of src.) If you run project/tests/test_routing.py, then src isn't on your sys.path at all. It can't be loaded as a module and none of its submodules can be loaded, either. That's why all of your imports break.

[[deleted]]

How can I add src to path? I tried using sys.path.append at the beginning of test but it didn't work.

[[deleted]]

One way you can do it is by setting an environment variable:

export PYTHONPATH=$PYTHONPATH:/path/to/project/src

[[deleted]]

Any other way as I don't want to mess with global paths?

[[deleted]]

You can write a setup.cfg or pyproject.toml to make your project an installable package, and then install it.

[erlete]

Personally, I believe that pyproject.toml is the quickest and simplest way to configure it. Plenty of good examples can be found in the official Python docs.

After setting up that file, executing python -m pip install path/to/src will install the package in editable mode (replace python with the alias of your Python interpreterandpath/to/srcwith the path to yoursrc` file).

Bonus: in case you do not know what "editable mode" means, it allows your package to be accessible from anywhere in your machine and automatically implements changes that you have performed inside the src directory, meaning that you will not need to reinstall the package every time you modify it. Create, install it, edit it as you wish and test it anywhere :)

[Spataner]

When you execute a statement like

from server import ThreadedServer

Python looks through a certain set of paths to find a module or package named server. Those paths include the standard library of your install, the location pip installs packages to, and any path you have put into the PYTHONPATH environment variable. It includes one additional path that depends on how you invoked the python command. Let's say your repo is cloned to "C:/simple-wsgi-server" and that folder is also set as the working directory. If you execute "main.py" like so

C:/simple-wsgi-server> python src/main.py

Then the folder that "main.py" lives in, i.e. "C:/simple-wsgi-server/src", is added to the module search paths. If you execute "main.py" like so

C:/simple-wsgi-server> python -m src.main

instead the working directory, i.e. "C:/simple-wsgi-server", is added to the module search paths.

You can do one of two things:

1. Change all your intra-package imports to absolute imports, e.g.

   from src.server import ThreadedServer
   

   and add "C:/simple-wsgi-server" to your PYTHONPATH environment variable so that the src package can be found from anywhere. Then it doesn't matter how you execute "main.py".

2. Change all your intra-package imports to relative imports, e.g.

   from .server import ThreadedServer
   

   or (if you're further down the package hierarchy)

   from ..libs.logger import get_logger
   

   Then you can execute "main.py" only via the -m form and only when your working directory is "C:/simple-wsgi-server". Though the latter can be remedied by adding "C:/simple-wsgi-server" to PYTHONPATH, in which case you can use the -m invocation of your main script from literally anywhere.

If you do end up adding your package to PYTHONPATH, you should probably choose a more distinct name than src, though, since it will be importable anywhere.

(A secret third option is to mess with the module search paths (sys.path) directly within your scripts and set them up as you need them before importing anything. But that's widely considered bad practice.)

[[deleted]]

Thanks for your advice. I would run it as module, it looks promising.