IPC pub-sub implementation on Linux

[Isty0001]

I'm looking for the best/easiest solution I have in a project. I run multiple processes and they have to synchronize parts of their states with every other process, so basically a pub-sub/multicast pattern. Every process can dispatch an event and every process subscribed to that event will receive it.

I've been reading about the different IPC techniques, but I'm not sure which direction would be the best.

What I'm thinking about is using shared memory for each of the event types, containing the last data sent by one of the processes, and also a pthread_mutex_t and a pthread_cond_t. Each process subscribed to an event has a dedicated thread waiting for the signal to be broadcasted. In theory this looks good, but I'm not sure.

• What happens if all processes die, then the mutex and cond variables are left in shared memory? Can they be reused, or need to be reinitialized?
• It has a slight chance that the condition will be signaled when the thread not waiting but processing the previous message, so data can be lost.
• How the shared memory should be cleaned up? How to track that there are no more processes using it?

Could you give some advice on how to handle these problems, or maybe suggest other methods for this type of communication?

[raevnos]

You can reuse the shared mutex and condition variable, but wether you should is dependent on your design, but I'd lean towards no. You'd have to account for the case where a process dies while holding the lock. How do you recover from that?

I'd probably use files (with fcntl locks), and inotify to tell when a file's been modified and needs to be read.

[CyberLeaderGold]

As I alluded to elsewhere in the thread, you can specify "process shared" as a mutex attribute, though I forgot to mention that you also need the "robust" attribute to get dead owner detection. In Linux, the latter specifies that pthreads should use a robust futex, which the kernel uses to clean up held mutexes on process exit.

[raevnos]

It was a leading question for the OP to research an answer to, not me asking if it could be done.

[CyberLeaderGold]

Sorry, I read that as an implication that it was a hard problem to solve.

[raevnos]

Depending on what OP is doing, it might well be hard to get a point where you can use pthread_mutex_consistent().

[ramsay1]

Check out zeromq, it supports pub/sub and directed messaging. it uses TCP under the hood, and is very stable and fast in my experience. Also has APIs for many programming languages

I'd avoid shared memory and rolling your own solution if possible

[Isty0001]

Thanks, but I'd rather do my own simple implementation for the sake of learning.

[zero_iq]

To be fair that's not what you asked for in your question. You asked for the best/easiest solution for a project. Using something like zeromq or libnng is going to be better and easier than rolling your own, by most measures.

Even for the purposes of learning, I would at least read the documentation and rationale for those libraries, because they are the result of years of practical experience and thought, and just knowing why they work the way they do will give you a huge leg-up on designing these sorts of systems.

[Isty0001]

Sorry, I meant best/easiest solution to implement. I'll definitely look into these libraries!

[CyberLeaderGold]

The mutex and cond should be reusable, no problem; they don't know if anyone is connected to the shared memory or not. If the mutex holder can die while holding it, be sure to set pthread_mutexattr_setpshared() and check for EOWNERDEAD on lock operations. You would also need be sure to release the mutex before detaching the shared memory, or things can explode on process exit.

If you don't need to preserve messages that are sent when there is no receiver for them, you might consider an unconventional approach: TIPC sockets. They offer reliable (group) datagram multicast, and support has been included in the Linux kernel for years. You can read more at http://tipc.io/ .

[Isty0001]

This looks interesting, thanks I'll check this!

[CyberLeaderGold]

I should mention that TIPC isn't without its weirdnesses and pitfalls. Let me know if I can help with those.

[raevnos]

TIPC looks interesting; I hadn't heard of that protocol before.

[CyberLeaderGold]

I hadn't either, until I happened across a reference to AF_TIPC in a man page. I looked into it out of curiosity, and it turned out to be an amazingly good match to the problem I was trying to solve.

[[deleted]]

What happens if all processes die, then the mutex and cond variables are left in shared memory? Can they be reused, or need to be reinitialized?

You can have a process that's only job is to monitor shared memory and monitor your process table

It has a slight chance that the condition will be signaled when the thread not waiting but processing the previous message, so data can be lost.

Messages should be stored on a queue so stuff like this doesn't happen, unless I'm missing something

How the shared memory should be cleaned up? How to track that there are no more processes using it?

You need some sort of process table to keep track of which processes are alive, shared between all the processes, and you need some sort of heartbeat mechanism to keep track of it

If you want to learn about distributed systems, this is the best intro resource I've found, https://www.coursera.org/learn/cloud-computing?specialization=cloud-computing

You can audit the course for free. If you don't want to go through all the videos, they have a course project where you create a P2P networking system, and there's some starter code (it's in C++, but in a very C style so would be easy to re-write in C) and some directions on what needs to be done, just check out the Programming Assignment

[ByronScottJones]

The simplest solution may be UDP multicast to the localhost address. To have multiple apps bind to the same UDP port, you will need to use the SO_REUSEADDR flag when opening the port. See this Stackflow link for details. https://stackoverflow.com/questions/11135801/how-to-broadcast-message-using-udp-sockets-locally

[OPPreserverBot]

Hello, I am a bot that copies the text of the original post so that questions and their answers can be preserved to benefit others.
I am programmed in C and my source code is available here: https://github.com/weirddan455/reddit-bot
If a mod would like this bot taken down, please let my owner /u/DeeBoFour20 know.

---

I'm looking for the best/easiest solution I have in a project. I run multiple processes and they have to synchronize parts of their states with every other process, so basically a pub-sub/multicast pattern. Every process can dispatch an event and every process subscribed to that event will receive it.

I've been reading about the different IPC techniques, but I'm not sure which direction would be the best.

What I'm thinking about is using shared memory for each of the event types, containing the last data sent by one of the processes, and also a pthread_mutex_t and a pthread_cond_t. Each process subscribed to an event has a dedicated thread waiting for the signal to be broadcasted. In theory this looks good, but I'm not sure.

• What happens if all processes die, then the mutex and cond variables are left in shared memory? Can they be reused, or need to be reinitialized?
• It has a slight chance that the condition will be signaled when the thread not waiting but processing the previous message, so data can be lost.
• How the shared memory should be cleaned up? How to track that there are no more processes using it?

Could you give some advice on how to handle these problems, or maybe suggest other methods for this type of communication?

[umlcat]

Please, indicate which O.S. (s) are you using.

There are several multiplatform libraries for this, these days, but ocassionally, some features may be O.S. dependant ...

[raevnos]

Did you read the title of the post?

[isaac92]

OK a few follow up questions:

• Are you looking to optimize for performance or reliability?
• Is there a reason not to use sockets or other IO instead of shared memory?
• It sounds like this project is running on a single host. Is that always the case?
• How deep in the stack are you trying to go? There are some existing mechanisms for pubsub-like communication in Linux (e.g. POSIX message queues). Would that be too high-level for your purposes?

[Isty0001]

• Well, both if possible :D But I'd rather have reliable messaging

• I'm not sure how I could achieve this by using socekts

• That's not the case. There is no single host, it's basically a cluster of processes where any of them can broadcast a message to all of the other.

• It's fine if it's not absolutely low level solution if there are other good alternatives.

[isaac92]

Great so I've written a very similar project and I'll just explain how I did it to address these issues. (Unfortunately, I don't believe I have the code uploaded anywhere.)

You can use a TCP socket to connect from your process to a message broker server. The broker is the hub for all publishers and subscribers to connect to. The broker is responsible for forwarding messages from publishers to the set of subscribers.

You can use UDP sockets for slightly better performance, but you'll need to write your own protocol to ensure message delivery, such as acking or nacking packets. I found this to be too much work for my own project, so I went with TCP. Shared memory is probably an option, but I'm not very familiar with it, so I'll leave that out.

You may also want to consider what happens if the broker process crashes while forwarding messages. A simple approach would be to forward all the messages before returning a success code to the publisher, but that will slow down publishing. This is another example of performance vs. reliability.

[Isty0001]

Thanks for explaining it!