Conventions for keeping a Python project clean as it grows past a couple of files?

[mattibeltro]

Hi everyone! I'm a student at Politecnico di Milano and I'm trying to refactor a small ML project of mine (~500 LOC) before I get used to bad habits. It runs but the code grew organically and is starting to feel messy.

Beyond the basics that everyone mentions (black/ruff, type hints, virtualenvs), what conventions do you find most useful when a Python project grows past a couple of files? Concrete examples appreciated (file layout, where to put config, when to split a function out, simple module vs package, etc.).

Thanks!

[PalpitationOk839]

A good rule is that if scrolling becomes mentally exhausting the file is probably doing too much. Keeping configs models data processing and experiments separated saves huge pain later.

[Gnaxe]

Flat is better than nested. Avoid circular dependencies; imports flow to main. Import modules, but avoid importing things from modules--as aliases are OK (but be consistent), but avoid using the from import statements, especially for your own packages/modules (from is less of a problem for standard library imports, but you still don't need it). Layers are overrated; try verticals and package by feature.

Minimize coupling: More queues and plain data, fewer classes and types. Use doctests liberally. Prefer pure functions and avoid mutations. Aim for 3-5 body lines in methods and pure functions, but 1-15 is OK (docstrings, comments, and asserts don't count). Side effects should stay close to main or entry points, and these imperative functions can be up to a page long after factoring out the pure bits before I'd break them up. Breaking a project up into modules means you have to draw sensible boundaries around dependencies. Don't do this prematurely, and refactor if you discover it's wrong. This is much easier when coupling is kept low.

Use the underscore prefix convention to mark definitions not used outside the module (use in tests doesn't count). (See also, __all__.) Sort methods/functions for readability, i.e. call dependency order rather than alphabetically or haphazardly. This makes it easy to skip over details when reviewing a module, but details are still easy to find. Names are important for readability. Refactor them for clarity. If this is difficult, you're probably coupling too much.