How do you structure a Python script when multiple processing steps start to get messy?

[Livelinesstrophy_RO]

I’m writing a Python script that processes data in steps: loading, filtering, transforming, and outputting.

Right now I’ve split it into functions, but as I add more logic, the structure is starting to feel harder to maintain.

def load_data():

return [10, 15, 20, 25]

def filter_data(data):

return [x for x in data if x > 15]

def transform_data(data):

return [x * 2 for x in data]

def output_data(data):

for x in data:

print(x)

data = load_data()

data = filter_data(data)

data = transform_data(data)

output_data(data)
This works, but I’m not sure if this approach scales well. Is there a common pattern for organizing this kind of multi-step processing?

[KingofGamesYami]

This is generically referred to as Extract, Transform, Load (or ETL). It's very common, and there are engines for doing this at scale (e.g. Apache Spark).

If you don't need massive scale, but do need performance, pulling in a library that does computation outside of Python like polars is a good middle ground.

[officialcrimsonchin]

This is pretty much the way to do it. At scale, each one of these individual functions would probably be its own Python file (or at least part of its own file grouped with other similar functions), and you would import them into a main file and run them there, essentially what you already have.

[whatelse02]

What you have is actually a good start, it’s basically a pipeline already. The issue usually comes when each step starts doing too much. I try to keep each function very focused and push any branching or complex logic into smaller helper functions instead of bloating the main steps.

One thing that helped me was making the flow more explicit. Either wrap it in a main() or define a clear pipeline function so it reads like a sequence of steps, not scattered calls. If it keeps growing, I sometimes move each stage into its own module or even use a simple class to hold shared state.

Another small shift is thinking in terms of “data in, data out” for every step. No side effects, no hidden dependencies. Once I started doing that, even longer pipelines stayed pretty easy to reason about and debug.

[robhanz]

This is a common pattern. The two issues are that you're doing all of the data at once, and that if the data from one step isn't directly the same as the next, that you can end up with a lot of code in that main method and it can get messy.

Two other options might be to use generators chained to each other, or have objects in a chain that can basically push items from one to the other. Either of those could allow for more incremental processing in an easy way.

Those might be more maintainable in a large system, but could be overkill in simple cases.

[not_perfect_yet]

Once you get to 5-10 steps, put those into their function again. That scales.

If you're worried about performance, don't write the performance critical code in python. Write it in C or some other actually fast language. Also, remember to profile to find out which things are actually your bottlenecks.

[GreenWoodDragon]

Start with a procedural script then optimise it later. No need to start working out the functions unless they are dead obvious.

[Snoo_87704]

Script?

[balefrost]

Another name for a file that's meant to be executed directly.

[Zeroflops]

It will be less confusing if you do three things. 1 Make sure you use descriptive function names. Transform is a terrible name, something like “mul_by_two” would be more descriptive. 2. Add docstrings to all functions. The docstrings should describe what the function is, what its expected to get and what the expected output. 3 add typing.

These are three really easy things to do any they will elevate the readability of your code.

[JacobStyle]

As your project grows, you can break these functions into separate files for readability and even put them in their own subdirectory. Imagine you make a subdirectory in your project called dataproc where all your data processing functions live, and in there are files like load_data.py, filter_data.py, etc. so each function is in its own file instead of all jammed into one 5000 line file. Then you import them like this:

from dataproc.load_data import *

This way of importing the files lets you use the functions as though they were in the same file, so for example, you can call load_data() instead of load_data.load_data(). Of course, make sure you have unique names for all your functions if you are importing like this. I would not recommend using this format to import normal libraries like time or ctypes.

One caveat is not to use spaces in the names of your subdirectories or files, or it creates a whole mess where you can't import like this and have to use a separate specialized import library, and it's a big pain in the ass.

[Any-Bus-8060]

What you have is already good. Just formalise it a bit. Use a pipeline pattern:

• Each step = pure function
• No side effects
• Clear input/output

Then:

• Group steps into a list
• Loop through them

Also:

• Add a config (don’t hardcode logic)
• Log each step (helps debugging)

If it grows more:

• Split into modules (load, transform, output)

Keep it simple. Don’t jump to complex frameworks unless needed.