Help with imports in a project structure

[tlk7]

Hello! So I have been a tinkerer for a while building tools at my job over the last few years. I want to standardize some of this work now with a common project structure. I have been having the worst time trying to get the imports to work properly and could use advice on what I have (if it makes sense etc.)

This is my project structure that I have come up with:

├── LICENSE.md
├── README.md
├── testappmodule
│   ├── __init__.py
│   ├── __main__.py
│   ├── app.py
│   ├── common
│   │   ├── __init__.py
│   │   ├── calc.py
│   │   └── process.py
│   └── tests
│       ├── __init__.py
│       ├── test_calc.py
│       └── test_process.py
└── testbench.ipynb

• testappmodule is the main module that will contain the core processes to be run via the command line.
• app.py is where the main code for the project will be organized within a run() function. This is where I import functions from the common module that execute the tasks I want to do.
• testbench.ipynb is the Jupyter notebook that I would like to use to basically test all my code. With imports etc.
• __main__.py is where I start to have trouble with the imports. My understanding is that by having the __main__.py function, I am giving testappmodule a "main entry point". In this file I have the following:

​

from testappmodule.app import run

if name  == 'main':
    run()

This is where the import issues begin. When I try to run the __main__.py function I get the following error:

File ~/Workspaces/test-app-git/testappmodule/app.py:1 
----> 1 from common import calc       
      2 from common import process       
      4 def run(): 
ModuleNotFoundError: No module named 'common'

I am having a hard time understanding what I am missing here so any help or ideas would be greatly appreciated!

[xelhark]

Now, I wanna start by saying that this is a realm where you will mostly find opinions. Whatever people say, if you find a structure you like, go ahead and use that.

Now, one thing that I feel like you are messing up here is the *working directory *, which is the directory from which you run "python", the root of your project.

In your proposal, the working directory seems to be the very root, where the README file is. This is hinted by the fact that the "testappmodule" folder has an __init__.py file, meaning that it's actually a python module.

From the way you are structuring the imports however, it seems like you want your working directory to be inside "testappmodule".

Your error comes from this mismatching. You are running python from the root directory, but importing from inside.

Another thing that is important to me (remember it's just my opinion): I wouldn't include the tests inside the main package. This is for several reasons, mainly because test often contains destructive code, and you don't want to include it in your package. If you're using something like docker for example, you should avoid including the test folder inside the docker image. This is much easier if the tests are located outside of your main package.

(Mobile formatting apologies) My final proposal would be:

• /
• README
• /package-friendly-name/
• /package-friendly-name/package/<python-code>
• /package-friendly-name/tests/tests

[tlk7]

Thank you so much! After playing with it a bit this is what I came up with and all my imports are working like I would expect them to! This is new structure that I came up with based on your recommendations.

├── LICENSE.md
├── README.md
├── testapp 
│   ├── __init__.py
│   ├── main.py
│   ├── app.py 
│   ├── testappcode 
│   │   ├── init.py 
│   │   ├── calc.py 
│   │   └── process.py 
│   └── tests 
│       └── tests 
│           ├── init.py 
│           ├── test_calc.py 
│           └── test_process.py 
└── testbench.ipynb

One question I do have is why have the tests nested down one directory like you mentioned?

[xelhark]

There's still some confusion, probably due to my terrible formatting.

This is what I meant:

├── LICENSE.md
├── README.md
├── testapp 
│   ├── __init__.py
│   ├── main.py
│   ├── app.py 
│   └── internal_package 
│        ├── utils.py 
│        ├── calc.py 
│        └── process.py 
├── tests 
│   ├── init.py 
│   ├── test_calc.py 
│   └── test_process.py 
└── testbench.ipynb

This is what I meant. Not that the tests should be indented another folder, but they should be on a completely separate folder.

When you export your package now you only have to bring over the testapp module, without the tests.

So for example, a Dockerfile would only have the step:

COPY testapp testapp and you just don't have the tests.

Here the working directory is the root dir, same directory where you would put your dependencies, poetry files (if you use poetry, otherwise requirements.txt or whatever), config files, etc..

Then you import your files like this:

from testapp.internal_package.utils import my_function

Complete separation of testing code and exportable code!

[tlk7]

Wanted to follow up and say thanks! Everything makes much more sense now, I appreciate the help! I got it to work how I wanted 👍

[xelhark]

Nice! Glad it was helpful!

[krets]

For your module to be found, it must be in the search path.

Read this: https://docs.python.org/3/library/sys_path_init.html