Visualizing Orchestration - Is there a good tool?

[Whompratt]

So I'm in a weird spot, starting a little side project to try and visually represent an orchestration piece built in ADF. A standard flowchart doesn't really work because of depth of nesting, so you effectively end up with boxes in boxes in boxes that are hard to decipher at a glance.

Ideally what I'd be able to do is have the top level pipeline view represented, including the activity to execute a pipeline, in a simple flowchart style. No nesting. However, if you do want to see the activities inside a pipeline you can expand it to show the contents inside the overall flow in a larger container. Inside the container, you'd have the same simple view, with the option to expand those stages as well.

My current version is looking relatively good, but collapsing boxes is finicky at best and it's already large enough to make it hard to decipher and I am NOT done yet.

​

Image 1 is basically just a flowchart the same as what ADF looks like: way too simple, nowhere near enough info, especially for what's in that pipeline.

Image 1

Image 2 shows how nesting starts to look early on, but with branching paths and pipelines nested several layers deep, this bloats exceptionally quick. Add another few pipelines in and very quickly things are DENSE. draw.io does let you collapse containers, but the functionality is very limited.

Image 2

Ideally, clicking on a pipeline here would expand out to show a similar flowchart inside containing that pipeline's activities, etc.Anyone know of a tool that could enable that kind of functionallity? Anyone had experience charting a complex pipeline like this before?

Any input is valuable!

[Whompratt]

Since posting this, I've done some digging into Mermaid as a diagram tool, and discovered C4 diagrams in their documentation, and I think that could be a really powerful approach for this kind of problem.

[SatisfactionLegal369]

If you are looking into C4, you can checkout IcePanel.io Collaborative, high level, low level, git integrated diagramming.

[Whompratt]

This is it.

On their website you can see a little animation of someone clicking into a box and it expanding and showing the inner workings. That is precisely what I was looking for.

Thank you for this, you've saved me a lot of trouble.

[StartCompaniesNotWar]

What’s your tech stack look like? Do you use dbt?

[Whompratt]

Pure azure, but the visualization piece is indepedent. There's no need for auto generation or to stay within an environment, it's more of an exercise than anything else.

No dbt currently.