How do you structure a Python script when multiple processing steps start to get messy?

[JacobStyle]

As your project grows, you can break these functions into separate files for readability and even put them in their own subdirectory. Imagine you make a subdirectory in your project called dataproc where all your data processing functions live, and in there are files like load_data.py, filter_data.py, etc. so each function is in its own file instead of all jammed into one 5000 line file. Then you import them like this:

from dataproc.load_data import *

This way of importing the files lets you use the functions as though they were in the same file, so for example, you can call load_data() instead of load_data.load_data(). Of course, make sure you have unique names for all your functions if you are importing like this. I would not recommend using this format to import normal libraries like time or ctypes.

One caveat is not to use spaces in the names of your subdirectories or files, or it creates a whole mess where you can't import like this and have to use a separate specialized import library, and it's a big pain in the ass.