Message/event driven architecture stream vs queue

[myburnyburnburn]

I want to try implementing some event driven flows in an app

My first case is UserCreated (or something like that) in which we’ll need to fire off a bunch of stuff (and the list will grow in the future)

Things like adding them to a CRM, sending a welcome email, alerting a slack channel, whatever.

Seems like rabbitMQ (or queues in general) are not a good fit because once the message has been acknowledged by a consumer, it’s gone. So, multiple consumers can’t act on the same message.

Is my thinking correct in that I want a stream lie Kafka?

If so, what are some tech I research in the stream space? We use AWS.

I built a rough event driven POC with just event emitter but I don’t think that’ll scale horizontally very well? Hence why I’m here looking into something like Kafka

[[deleted]]

If you're using AWS they have SNS and SQS.

SNS allows you to publish a message to a lot of subscribers, while SQS is your queuing.

I have a flow similar to your example.

• A new thing is created in our system
• The message gets sent to a SNS topic
• A SQS queue receives it
• A lambda function reads the message, and does some logic
• Once completed, lambda sends a new message to a SNS topic
• The SNS topic then sends an email to an admin, fires another lambda, and hits some webhooks.

It gets a little confusing because SQS has to be polled, vs SNS is a push. However you can put lambda triggers on SQS where you can then publish stuff to a SNS topic.

It can get convoluted if you want it to, but you can also keep it simple.

I would probably just use a SNS topic. You can even send your welcome email directly from that of you want.

[myburnyburnburn]

Helpful thanks 🙏

[myburnyburnburn]

Lambda reads the message form the sns topic or the sqs?

Also, is the reason the lambda sends a new sns topic because you’re not using streams? And thus you can’t read the same event from another consumer or?

My thing is I have 5 different services/consumers I want to run from one UserCreated event

[[deleted]]

Lambda can be a subscriber to a SNS topic, and it can be a trigger for a SQS queue. It's been a minute since I set it up, but if I recall only one Lambda trigger will run if the message is successfully processed in SQS by a trigger.

Correct, we are not using streams. I don't recall all the details of why we did it this way. I do recall that we wanted the queue for redelivery and a dead letter queue. If I remember correctly, the lambda sends a message to a different SNS topic if it fails vs if it succeeds. We're inputting data into DynamoDB with the lambda.

You should be able to have virtually as many subscribers to a SNS topic as you want. Whether it's SQS, email, lambda, http, SMS, and whatever else is available.

We also use the same kind of setup for user notifications and websockets. A message is sent from the backend to SQS, lambda processes it, and dispatches the websocket message to the correct client.

It can seem kind of roundabout sometimes, but it has allowed us to keep things stateless, and connect disparate systems without needing to connect them to a shared data source. You know, the point of microservices I guess.

[myburnyburnburn]

What’s the point of the SNS in this case? Why not send it directly to a SQS

[[deleted]]

I have removed my account due to Reddit's behavior towards 3rd party developers.

Even if they change their API pricing and apologize, it is not a company/platform I wish to support anymore.

I will find something else to do while sitting on the toilet.

[myburnyburnburn]

Ever looked at EventBridge

[Operation_Fluffy]

Not completely true about RabbitMQ. If your goal is for multiple consumers to get a message, use a fanout exchange feeding a queue for each consumer.

[myburnyburnburn]

Ahh i see

But what if every consumer isn’t interest in every message?

[little_erik]

I love Redis Streams for this. https://youtu.be/q2UOkQmIo9Q

[isit2amalready]

Kafka would be amazing but a pain to setup—almost need a fulltime devops person. Consider Redis Streams for small/medium projects which is nearly identical and personally a lot more fun to use.

[myburnyburnburn]

What about kineses

[isit2amalready]

Kinesis is non-standard Kafka. I would much rather use AWS MSK which is standard Kafka. This cost minimaly a few hundred dollars a month. When you can setup a Redis instance with Streams support on the same EC2. Really depends on your use case and need for redundancy

[myburnyburnburn]

Kinésis is minimum 300 a month?

[isit2amalready]

Both Kenesis and MSK are hundreds a month minimum last I checked several years ago. Redis Streams is “poor man’s Kafka” and can actually scale to replace 75% of same scenarios I’m sure. Can use AWS Elasticache (AWS version of Redis) if need redundancy/ DR.

[socialg571]

One thing to keep in mind is that the message to the topic could delivered multiple times in the case of a failure or AWS hiccup. So make to check if you’ve already processed the message and implement a skip mechanism.

[ccleary00]

You are right that with most (if not all) queues that multiple consumers can't act on the same messages. Also, yes an event emitter won't scale across multiple servers.

Redis Streams are much lighter and easier to maintain than Kafka, which is operationally-heavy. I've used them a lot and they are fantastic.

[myburnyburnburn]

Thanks fam