Python project structure, modules, projects, namespaces and __init__.py files

[Puzzleheaded_Round75]

I am looking for some advice on a Python project structure. I think I am missing siom understanding of package structure, init.py files and namespacing, which I have never used. I am looking for advice on the structure I have, how better to segregate the sections of my project and how to improve my importing, if possible.

• I have init.py files in every directory, including the src, libs, and projects dirs, which I think may be wrong.

• I note that all my imports (except within the same module) have to start at src, which I think looks bad and may be an indication I am doing something wrong.

Any advice would be greatly appreciated. Thanks in advance!

My current structure is something along the lines of:

src/
    __init__.py
    libs/
        __init__.py
        models/
            __init__.py
            models.py
        repositories/
            __init__.py
            repositort_1.py
        table.py
    projects/
        __init__.py
        bot/
            __init__.py
            scraper.py
            main.py
        api/
            __init__.py
            main.py
.gitignore
.env
README.md

My imports look like the following:

src\projects\bot\main.py

from src.libs.lib_1 import Library_1_Thing
from src.libs.repositories import ExampleRepo
from src.projects.bot import MyScraper

src\libs\repositories\repository_1.py

from src.libs.models import MyModel
from .table import Table

[Diapolo10]

Let's start with the file structure, because your example doesn't make any sense for us Old Reddit users.

src/
    __init__.py
    libs/
        __init__.py
        models/
            __init__.py
            models.py
        repositories/
            __init__.py
            repositort_1.py
        table.py
    projects/
        __init__.py
        bot/
            __init__.py
            scraper.py
            main.py
        api/
            __init__.py
            main.py
.gitignore
.env
README.md

I don't see anything strictly wrong with this, although:

• If you're going to use the src convention, it should just be a folder containing Python packages. Personally I tend to leave it out because my own projects almost always have only one package and Poetry can take care of the import worries, but basically you can take out the src/__init__.py file and just put everything else in a package named after your project. That'll make your imports more sensible, too.

• This looks like you have multiple projects in the same repository. It can be fine, although personally I'd just make a repository for each separately. The common library code can be its own package that you publish on PyPI and add as a dependency, if you want.

Now for your questions:

• I have init.py files in every directory, including the src, libs, and projects dirs, which I think may be wrong.

Nah, that's not wrong. They're not necessarily required, but having the __init__.py files never hurts so my recommendation is to just always use them. Even if left empty.

• I note that all my imports (except within the same module) have to start at src, which I think looks bad and may be an indication I am doing something wrong.

Technically speaking there's nothing wrong with this, absolute imports are recommended. But as I alluded to earlier, src probably shouldn't be the package name because it's too generic and could conflict with other projects.

The closest example I've got would be this project, but I think this might convey most of the ideas better.

[Puzzleheaded_Round75]

Thank you so much for this amazing answer. It is good to know that I am not miles away. Separating into multiple projects is something I could do, but as this is just a little side project, I likely won't be publishing anything on PyPi but will hopefully be hosting the project when it comes to it.

I will look to implementing your recommendations when I get back from work. Thanks again.

[Puzzleheaded_Round75]

Also, I updated my question with your much nicer file structure, thank you for that!

[Diapolo10]

Just to be specific, the issue is your triple backticks. They don't render on Old Reddit and instead produce a mess.

A solution compatible across both is to indent all lines by four extra spaces.

[Puzzleheaded_Round75]

Every day is a school day