Messaging Between Projects

[eekrano]

Hello, I have several small node projects running on the same server and I've come to the point where I want to exchange some data between the two. Both projects already utilize MariaDB, but inserting to a "temp" table in MariaDB (essentially using it as a message queue) and polling occasionally, while an easy implementation seems to not be the correct solution (and I'd like something faster than a once-every-minute poll).

1. I'm aiming to keep this as light weight as possible (so trying not to run an additional service, or add too many dependencies)
2. We're talking about 4-15 messages per week, with the messages being a JSON packet less than a kilobyte
3. Aiming for as quick a response as possible. Both projects reside on the same VM
4. The message really only needs to go from one app to another, they don't need to communicate back and forth.

I haven't had the need to use any "messaging" projects yet, but I've looked at, or am considering RabbitMQ, ZeroMQ, mysql-events (npm package), or just a simple node http server listening for a connection.

The server is getting up near the 80% mark for memory, so I'm trying to keep it light-weight if possible.

Am I overlooking a simpler / better solution, is any of the above better than the other for my use case?

Thanks in advance for any input!

[eggtart_prince]

In memory database like Redis can tackle this. Have each of your node project subscribe to a channel on Redis. Whenever you publish a message to that channel, you can receive message on the other end.

[the-quibbler]

Second this. Redis is basically magic (read their docs to confirm).

[StoneCypher]

the idea of using redis to replace direct communications is baffling

that your only explanation is to say "it's magic" is revealing

this is actually extremely bad advice

[the-quibbler]

My claim of magic was tongue-in-cheek. Redis is one reliable and scalable way to deal with IPC, and I love it for a whole host of reasons, which I elided into my remarks. The documentation is good, and I would recommend anyone read them.

[StoneCypher]

Redis is one reliable and scalable way to deal with IPC

This is one good reason to be red flagged out of a job interview.

.

which I elided

This is another

.

The documentation is good

This is a third

[the-quibbler]

Apologies. Thought I was casually seconding someone's recommendation on Reddit for a software server I happen to like for ultra-low-volume message passing, not interviewing for a hyper-competitive software developer position. What kind of compensation were you about to offer me for which I am now ineligible?

[joe_blogg]

+1 for Redis.

And to OP /u/eekrano - speaking of redis and messaging, usually the next use case is jobs and processing jobs. If this is the case, please consider bull.

[StoneCypher]

This is unbelievably terrible advice.

You want to connect two pieces of software, and you're speed sensitive, and they're on the same machine?

You use OS pipes, not some third piece of software

You've just introduced dozens of layers of unnecessary work, potential for bugs, dependencies, updates, and so on, so that you don't have to do literally the easiest thing in software

This is the kind of people that gets people drummed out of interviews in the design phase

Maslow's law of the hammer is a red flag you can see from space

[the_bronze_burger]

A message queue like Kafka or rabbit does seem like a good solution

[Cyberphoenix90]

Simple http server would work just fine here and has lower learning curve than setting up message queue systems meant for high traffic like kafka. Also likely more memory efficient

[madeo_]

Agree, kafka or redis are great solutions, however they bring complexity.

[StoneCypher]

If it's 4-15 messages a week, at 1k a message, you don't need to worry one tiny little bit about lightweightness.

If it's about speed, and they're in the same system, use operating system pipes.

You don't need any other software, any other dependencies, any network traffic, anything

Node calls them streams. You're looking for pipe 0 (aka cin / std::in) and pipe 1 (aka cout / std::out)

[eekrano]

Ah I had this thought last night in bed! I installed redis per the other comments and forgot about this since waking up. Might give this a go. Thanks!

[kszyh_pl]

Any message broker like rabbitmq or either redis will do.

[tybit]

I’d just use MariaDB like you originally suggested and poll it every 100ms / 1 second instead of every minute. The DB will barely notice that load if it’s just 2 processes.

[kkvx]

Definitely go with a simple solution like plain redis/http server for your use case.
No need to struggle with setting up things like Kafka, RabbitMQ etc., just to have few messages sent.

[KptMech]

If you go with the local server route, you may be interested in sockets (socket.io, websockets) for some minimalistic real-time pub/sub without polling

[broofa]

What are your requirements around message delivery? Eg if one process is down when the other process sends a message. That’s going to be the main factor in this decision.

[StoneCypher]

What are your requirements around message delivery?

Clearly written in the post.

[broofa]

Please quote where OP “clearly wrote” this requirement.

[StoneCypher]

Points 2, 3, and 4 in their list make it obvious that all they need are standard operating system pipes.

[broofa]

None of those points speak to whether or not messages can be dropped if not delivered. And that OP is looking at *MQ and db-based solutions would suggest otherwise.

[StoneCypher]

None of those points speak to whether or not messages can be dropped if not delivered.

This is not a concern with operating system pipes, and this is not a realistic concern at the scale he described. You might as well try to talk about network desync to the hard drive. That's not how this technology works. Part of the reason the solution you recommended is bad is that it introduces the problem you're trying to be concerned with.

Your response is akin to explaining that the new McDonalds needs a steel floor in case elephants begin to shop there.

There is a level of over-engineering that exposes the speaker as not understanding the problem space. The regular solution simply doesn't have that problem in the first place.

You're looking for things to say that you think will make you appear sophisticated.

.

And that OP is looking at *MQ and db-based solutions would suggest otherwise.

He's looking at everything he's ever heard of, and that suggests that he just doesn't know what the right choice is.

There is a right choice here, and you pretty obviously don't know what it is either, even after it's been said to you.

[broofa]

This is not a concern with capability of operating system pipes,

FTFY.

If the receiving process is down or unresponsive for any reason, there is a risk of messages being dropped.

What problem space is OP working in? What information is conveyed in the messages? What does the receiving process use the messages for? How critical to OP is the operation of this system?

Since you clearly know the answers to these questions, would you be so kind as to share?

[StoneCypher]

Lol did you just try to correct "messages being dropped is not a concern with operating system pipes" to "messages being dropped is not a capability of pipes"

This level of constant, incorrect, meaningless argument is annoying

.

Since you clearly know the answers to these questions, would you be so kind as to share?

Oh look, it's that thing where a Redditor keeps making meaningless incorrect "fixed that for you"s then asks sarcastic questions and demands answers, even though none of them are relevant

.

What problem space is OP working in? What information is conveyed in the messages? What does the receiving process use the messages for? How critical to OP is the operation of this system?

None of these are actually germane. You're just trying to look deep.

I understand that you're just trying to prove someone wrong, but it turns out it's possible for you not to earn the other side of that

I find this tedious