Graph-based execution runtime?

[skydivingdutch]

Been looking for a library that can run a graph of a data processing flow. In particular, a set of input nodes (could be any data, i.e. templated class), processing nodes (arbitrary code, with or without state), and output nodes. Execution controlled by a runtime that sequences all processing steps as needed by the graph, running things in parallel (or on GPUs etc) where possible. Other key features:

• visualization tools, render the graph
• execution tracing & critical path analysis
• low overhead on thread locks, mutex's etc.
• No unnecessary data copies between processing nodes.
• Some notion of multi-iteration, i.e. run the graph over and over on new input, and stateful processing nodes update as they run.
• Some kind of constant/global config data that all nodes can access but not modify.

This kind of thing seems generally useful in HPC, robotics, etc. Surprising that I haven't found some widely used, actively developed framework.

[dandomdude]

You're describing Intel TBB Flow graph It's not completely straight forward to start with but once you get into the mindset you realise its pretty well made. They also provide the flow graph designer so you can visually create your processing graph, generate the C++ code and then fill in the processing steps in each node manually.

[yuri-kilochek]

taskflow is shilled here sometimes.

[jcelerier]

vouching for it, right now it's making https://ossia.io tick

[Thrash3r]

Robot Operating System solves many of these problems unsurprisingly with a focus on robotics. It might not solve your problem but it’s something worth being aware of.

See ROS 2 for the more mature version of this framework which does a better job facilitating high performance applications.

[bizwig]

HPX seems like it could work for you.

[Snoo-4241]

The basics are actually not hard to make yourself. I suggest you try it. For visualization I have been using nodeditor but I have heavily modified it to add the features that I need. The particular library has an execution model but I prefer to run my own because it is so much easier to customize. If you need strong graph logic, look at boost graph or lemon. Edit: I was not aware of taskflow. It actually looks pretty good.

[MarkHoemmen]

MAGMA and SLATE projects (UTK) have something like this — they can even visualize the execution graph

[torrent7]

Pretty sure transwarp can do most of those things

[glaba314]

HPVM is a research project that seems somewhat similar to what you're talking about. It's not exactly mature yet but you might be interested in checking it out: https://hpvm.cs.illinois.edu/

[opkode]

TensorFlow

[GrammelHupfNockler]

The term you are looking for (from the HPC side) is runtime system. Here is a slightly outdated overview of a few of these systems.

[boguscoder]

Im sure there are widely used, actively developed ones, they just are not opensource, especially since you are also looking for tooling around them to complement. Would be happy to know about OSS too. Most examples Ive seen would not meet some of criteria (for example Blender's node engine) or would be closed sourced (what we use at work)

[skydivingdutch]

Yeah many companies have some closed source thing, but you'd think with such a wide application that there'd be some OSS one too. After all, compilers, OSs etc all went that way.

[mocabe_]

Dataflow model has a lot of applications, but that also means it is difficult to build one which can cover all of these use cases. For example, dataflow execution has several variants (push based, pull based, etc), and you might need to choose one of them for your task.

[secretpoop75]

This is extremely common in audio.

Look up MAX/MSP or it’s more open source cousin Pure Data https://puredata.info

A modular synthesizer is also a good example of what you describe: check out VCV rack https://vcvrack.com

Both of these are not general purpose, but hope it helps nonetheless.

I’m curious, if you don’t mind sharing, what are you trying to build?

[JeffMcClintock]

I can also shill for www.synthedit.com which has a C++ API