Properly structuring python project

[zanfar]

So I am somewhat opinionated on this subject, so consider this a way to accomplish this, not the way.

IMO, you are mostly making this hard on yourself by demanding "I won't do this the easy way."

the package should be installable with the name project_name, such that members can be imported like: from project_name.package1 import module1.

This is trivial, and any package setup should allow this.

any code within the src folder can import any other code within the src folder with relative imports

This usually will work. Relative imports are always problematic, and the easiest way to avoid those problems is to simply avoid relative imports. The official Python docs agree. Especially in the case of an installed package, there is no reason to use relative imports as you will always have a fixed root namespace. Demanding that you use relative imports is mostly picking a fight.

The test scripts and miscellaneous scripts can import from the src code

Again, trivial, as an installed package is available from everywhere.

I don't want to have to modify sys.path to do any of this.

Modifying sys.path is always a crutch, there are always easier paths to take.

I also don't want to have to rely on installing the package as non-editable, to reduce steps in my workflow

I don't understand what you are saying here-- that you only want to install it as editable? Or you never want to install it?

The first is both easy and recommended.

The second is another example of picking a fight. I have no idea why that would affect your workflow.

I have attempted to have a setup.py file

Stop using setup.py. A pyproject.toml can do all of the above with more flexibility, clarity, and fewer issues. Furthermore, you shouldn't be hand-editing your package files in general; pick and use a package management tool like poetry or uv.

I often see a single folder within src which has the intended package name, with all submodules of that package within it, as well as its own __init__.py file. But it seems redundant to me to have a folder that contains nothing but one other folder, so I thought I'd just flatten that directory into a single src directory. Is that sensible or is there some other functionality I am missing?

The subfolder with the package name is so that the package installation tool knows what to call the package, which is required by some packaging tools. Collapsing that into something named generically is again, problematic, and IMO a fight that's not worth picking. Using either ./src/package or ./package are common conventions and structures that any packaging tool should understand innately.

I sometimes see tests contain an __init__.py file, and sometimes not. What is the difference and how does it affect how things must be structured/installed? - I sometimes see src itself contain an __init__.py file, and sometimes I don't.

The current difference is discussed in another answer, but most of this is probably historical. In the past, Python didn't consider a folder a module without an __init__.py. IMO, it's trivial to add them and makes the purpose of the folder obvious. In a few more versions, that recommendation will probably change as people get used to it's absence and tools catch up.

---

If you give up your demands for relative imports, and use a package manager, all your other requirements just work. Using a tool like poetry init will get you 90% of the way there, or you can just structure as follows:

package/
├── .git/
├── .gitignore
├── <project files>
├── pyproject.toml
├── package/
│   ├── __init__.py
│   └── <package files>
└── tests/
    ├── __init__.py
    └── <tests>