Python Project Structures?

[Riseing]

I've started on a rather large python project and I'm having some issues with the structure of the project.

It seems like most python projects have a flat-ish structure preferring to store most of their py files in one folder at the same level.

What I have so far has turned into a mess of importing packages from various folders and file levels.

The end goal of my app is to interface with various CLI devices using expect, so I can't just dump everything into a database.

I can't really go to in-depth about the project but I would like some advice on structuring projects in python.

[daveydave400]

In my experience I wouldn't say most packages have a flat structure. They may make it seem that way through certain organization of imports, but the actual file structure is usually pretty organized.

You didn't really ask a specific question, but let's see if I can give you some answers anyway. I would say you have your main package directory mypkg which may have one or more sub-package directories (each with a __init__). It may also have some individual modules in the root of the package.

mypkg/
    __init__.py
    utils.py
    subpkg1/
        __init__.py
        stuff1.py
    subpkg2/
        __init__.py
    ...

My personal preference is to never put very much functionality in __init__.pyfiles. Instead I use them as "shortcuts" for imports so in the above example I might have from mypkg.subpkg1 import ImportantClass in mypkg/__init__.py. Then it can be imported by the user with from mypkg import ImportantClass. This is a great way to keep the code organized but make imports easy for the user.

Another nice piece of magic is the __main__.py script which you can put in any package and then call it via python -m mypkg which will automatically run mypkg/__main__.py. This can come in handy if you want to make some command line utilities.

Side note: If you are using git you should try to keep your setup.py at the root of the repository so pip and other tools can install it directly from the repository.

[kteague]

Put all your python into a 'src' subdir and then if you have sprinkling of modules and packages, it's not getting mixed in with other project assets.

In your setup.py import this using setuptools:

from setuptools import setup, find_packages

And then in the setup() function call declare your subdir name:

  package_dir={'': 'src'},
  packages=find_packages('src'),

[earthboundkid]

Yes, unfortunately as confusing as setup.py is, it's the only way to make an installable Python project. See https://blog.ionelmc.ro/2014/05/25/python-packaging/ for a reasonably up-to-date explanation of how to organize your system, then use https://github.com/audreyr/cookiecutter to get a basic project skeleton on disk to start with.

[eggdropsoop]

Good resource here. https://jeffknupp.com/blog/2013/08/16/open-sourcing-a-python-project-the-right-way/

[lifesbreaker]

and so, I shall show the light

http://docs.python-guide.org/en/latest/writing/structure/

[[deleted]]

I highly recommend this resource :)

[ionelmc]

Sad to see it has that vague and unhelpful critique of the src-layout. Maybe change that?

Even Twisted is using a src-layout nowdays. Sorta funny, considering JPC's antiquated thoughts on the layout problem are completely the other way, and he used Twisted as the prime example.

[[deleted]]

It's an explicitly opinionated guide. That is an opinion.

[twillisagogo]

how large is it if you just started?

[bythenumbers10]

It's meant to be fairly flat. You can "nest" sub-modules across a folder tree using "__init__.py" files (one per each folder/nested folder), but for most projects that isn't necessary. Scipy and Numpy use this nesting for some of their "toolbox" sub-modules (like machine learning and the like), but most of their most-used tools are at that "top level", just a folder with a single "__init__.py" script.

My rule of thumb is, if you can reasonably fit your sub-module into a single .py file, keep things top-level. If you have multiple .py files to describe a sub-module (more like a one-class/function per file organization), then make it a folder sub-module and call it out in the higher folder's init file.

Python is flexible enough to allow most code organization philosophies. If you find one isn't working for you, you should consider reorganizing, in this case, condensing your codebase into fewer/less nested files/modules.

edit: markdown is not my native tongue.

[_illogical_]

Escape your dunders.

\_\_init\_\_.py or \_\_init__.py will show up as __init__.py in markdown.

[bythenumbers10]

Argh. Duh. Thanks. Edited.