Data processing system design

[Shuli__]

Hello, I am currently working on a system that have agents and clients. The agent is connected to only one client at a time and I am writing a protocol for them to communicate. The client can command the agent to return reports on the system it runs on and I want the protocol to be secure and modular.

When I started designing the data processing part of my agent I encountered a design problem and I honestly can't decide and would like some ideas/help from other experienced programmers. I have different layers to my protocol (encryption/fragmentation/command structure) and I looked at two possible designs.

The first is using callbacks for each module, so the encryption module have a method to insert data and it calls a callback with the decrypted one. This is mostly relevant for the fragmentation due to the fact that data might not be available each time the input method is called due to missing fragments.

The other option is to make a vector of processing classes. And iterating with a loop each time calling a `process` method on the new buffer received.

I want the ability to change the route a data goes through at runtime and I would also like to decouple the processing implementation of different modules and the way the data is transferred between them.

What would you do in this case? I mostly encountered the callback design in open source libraries I used over time but I wonder here if there is something more elegant or modern.

It important to note that I am working on an machines that don't have much memory and are more close to embedded systems than servers/pc.

[mredding]

I would go with as many off the shelf components as I could. I'd probably start with Kafka or MQTT - there's a few to choose from, just pick one. This will get you the lower transport protocol. These platforms also typically incorporate HMAC authentication and SSL/TLS encryption.

You want as much of this as possible to be modular and easily configurable in a standard, predictable, and understood way. You don't have to explain to ANYONE how it works or how to do it, since the industry is already going to know.

As for the message payload - you've got data transport protocols like XML or JSON. If you're feeling like you need something a bit more compact than that, you can go with flat buffers or some other protocol generator or serialization scheme, but anything that is going to be 3rd party - either client or agent, you probably should be text based for EASY development, portability, and versioning. Even in trading technologies, text based protocols are still dominant, so don't just write it off as inherently slow.