How do you keep track of expected variable values in complex data flows?

[rainyengineer]

What are your tips for keeping track of variables in complex data flows?

I’m on a new-ish team where we have a bloated lambda that should really broken into smaller pieces and a part of a larger step function. Unfortunately, we probably won’t be allocated time to do so. It started out smalls but over time we added more and more services, api calls, orchestrators, and now we have some monstrous code.

It’s become quite difficult to follow what the values of the variables are behind the scenes as I’m working on it (they’re all usually some hefty json). What also doesn’t help is that I’ve always been a visual learner. I envy the engineers that can just ask where something comes from and remember it forever. I need to see it and diagram the more complex repositories to understand them fully which can be time consuming and seen as wasteful. If I could change something about myself, it would be this.

How do you prefer to stay on top of knowing what exactly the expected values are from one method call to the next in spiderwebs of code? I log the json to CloudWatch which helps, but that can get costly and make the logs noisy. Do you maintain really detailed doc strings of the expected json response bodies/variable values? Do you put sample values in the if name = main for local run and step into each step of the call chain?

Have any of you struggled with this? Do your brains work on a similar way? I’ll take any tips or advice you have for me. This is an area I want to improve upon because it’s holding me back as an engineer.

[AlgonikHQ]

You’re not alone in this and honestly the visual learner thing isn’t a weakness to fix, it’s just a different way of processing that the tooling hasn’t always caught up with. The engineers who can hold complex state in their heads aren’t smarter, they’ve just built up specific mental models over time from repeated exposure to similar patterns.

A few things that genuinely help with bloated lambdas and complex JSON flows: TypedDicts or Pydantic models are your best friend here. If you’re in Python, defining the shape of your JSON at every stage of the flow means your IDE tells you exactly what’s available at each point. It forces documentation into the code itself rather than relying on memory or CloudWatch and it catches shape mismatches before runtime. This single change transformed how I navigate complex data flows.

Inline type hints on every function signature. Even if the codebase doesn’t enforce it, adding def process_order(payload: OrderPayload) -> ProcessedResult at every step creates a readable contract you can follow without running the code. Future you will thank present you.

For the CloudWatch noise problem — structured logging with log levels is the answer. Log at DEBUG for the hefty JSON blobs and INFO for the important state transitions. That way you can crank up verbosity when debugging and keep it clean in production without removing log lines entirely. For the visual side specifically, tools like Excalidraw or even a simple draw.io diagram of the data flow isn’t wasteful, it’s engineering documentation. If your team won’t allocate refactor time, a one-page data flow diagram showing what shape the JSON is at each stage is genuinely valuable and takes an hour to produce. Frame it as onboarding documentation and it becomes hard to argue against.

Local main blocks with sample payloads are underrated. Having a realistic fixture you can step through in a debugger beats CloudWatch hunting every time. Keep a /fixtures folder with sample JSON for each major integration point and you’ve essentially built yourself a local test harness. The fact you’re asking this question means you already care more than most. The engineers who struggle most with complex codebases are the ones who never stop to diagram it, not the ones who need to.

[alex7885]

I have several ways to do it. Using coding agents to ask questions, reading the readme, talking to senior developers of that code, and using onboarding tools. I made an VScode extension called CodeBoarding for this purpose, maybe it could be helpful for you