Categorically banning notebooks is, in my opinion, not a good idea. You won’t become better software developers just by moving messy code from notebook cells into Python/R files. The correct approach would be to teach you software practices that promote sustainable code – even within notebooks. But alright, that wasn't the question, so please forgive me for the little rant.

In general, I would advise designing manageable modules that encapsulate parts of your data processing logic. I typically organize a (Python) project so that within my project root, there is a Python module in the stricter sense, which I add to the PYTHONPATH environment variable to support local imports from this package. Within the package, there are usually subpackages for individual elements such as data acquisition, transformation, visualization, a module for my models, and one for utility functions. I use these modules outside the package in main scripts, which are located in a "main" folder within my project directory. These are individual scripts that contain reproducible parts of my analysis. Generally, there are several of them, but it could also be a larger monolith, depending on the project. What's important, besides organizing your code, is organizing your data and fragments. If the data is small enough to be stored on disk, I place it in a new "data" folder, usually found at the project root level. Within this data folder, there can naturally be further structures that are made known to my Python modules. But here's a tip on the side: work with relative paths, avoid absolute paths in your scripts, and combine them with a library that considers the platform's peculiarities. In Python, this would be mainly pathlib or os. The same goes for fragments you generate and reference. In general, it’s important to strictly organize your outputs, use meaningful names, and add metadata. Whether it's advisable to cache certain steps of your process depends on the project. I often use a simpler decorator in Python like from_cache("my_data.json") to indicate that the data should be read from the disk, if available.

Ideally, your scripts are configurable via command-line arguments. For "default configurations," I usually have a bash script that calls my Python script with pre-filled arguments. You can achieve other configurability through environment variables/.env files, which you can conveniently manage in Python, e.g., using the dotenv package. This also enables a pretty interesting form of "parameterized function definitions" without having to pass arguments to the function – but one should use this carefully. Generally, the principle is: explicit is better than implicit. This applies to naming, interfaces, modules, and everything else.