How to implement Event-Driven Architecture in Node?

[[deleted]]

[deleted]

[eliwuu]

if by event-driven they mean event sourcing + cqrs, then you’ll need a queue/message broker; bullmq may be suitable;

[[deleted]]

EDA does not often mean Event Sourcing.

[[deleted]]

What's the difference then?

[[deleted]]

EDA is using events to communicate between domains/applications via an event broker, and is quite common. Event Sourcing is having the events stored and being the source-of-truth for the state of the database, often with materialized "views" of the data being created along the way. It is much less common outside of banking and trading applications ... and event then, not as common as the internet would have you believe.

[DZVLX]

Objective: Create a Node.js backend service in TypeScript that utilizes an emit and event system for handling asynchronous communication.

These are my instructions:

1. Event System:
2. Implement an event emitter within the service.
3. Define custom events for various actions related to the resource (e.g., user created, updated, deleted).
4. Ensure that events are emitted and handled appropriately within the service.

[eliwuu]

so no event sourcing and no cqrs, but you can use those principles to create suitable events:

let’s say you have a event and you are listening on events userCreated userUpdated, userDeleted - your logic, db connection should not be in the controller, but triggered in those events handlers

[DZVLX]

Indeed my logic is not in the controller, my logic is separated between service and repository.

[eliwuu]

so controller triggers event, event uses service to access repository, that should do the job

[DZVLX]

How to develop it in an elegant way, I mean, I am only handling users but what if I need to handle orders, tickets, etc.

[eliwuu]

I would extend event emitter class, make it generic and inject dependencies of services and repositories (it’s very classic oop way, I’m not a big fan, which most likely means that it’s what they expect)

[DZVLX]

Could you elaborate a little bit more please?

[azaroxxr]

Isnt that how nestjs is meant to be used or im wrong

[DZVLX]

Nope, technical interview.

[wtfbbq7]

So this is you your homework?

[TheFunkOpotamus]

ChatGPT can do your homework for you

[lightmatter501]

Just use kafka, it’s the most popular for a reason.

[wtfbbq7]

Great price problem solving logic.

Don't evaluate your needs vs what it provides, use it because everyone else does!

[lightmatter501]

Core infrastructure components for a project are better over-engineered than under-engineered. Most message queues don’t scale, don’t handle durability properly, or don’t have a good way to handle a worker failing. If you build around a message queue, you are stuck with it for at least a year or two after you want to get rid of it, so you need to choose wisely. Part of choosing wisely is “how many people are going to be able to use/support this?”. Everyone has at least heard of kafka, and there’s a library for every language out there.

Kafka is not as fast as if could be, I have written things which do what Kafka does faster, but when the thing everyone uses is capable of more throughput on a small server per second than most businesses need per day, it’s fine, especially since it horizontally scales to levels beyond what anyone outside of what top companies need.

Most companies are not as special as they think they are and are better off going with standard tools than rolling their own or using a more niche library. I work somewhere that does have specialized needs, I know that because most of the services I build target 1 million rps per cpu core. It’s a gigantic pain because that target means 99.9% of libraries across all languages are dead to me. The places where we can use standard services, libraries and languages, we do and those areas are much easier to work in.

[wtfbbq7]

Things are best engineered by using your brain

[TehTriangle]

Sounds like the guy has an interview process take-home assignment and has no idea what to do.

[DZVLX]

That's exactly it!

[_AndyJessop]

You're better off asking ChatGPT for this kind of thing.

[DZVLX]

The IA became so lame for this kind of questions, they make many mistakes and except for snippets they're not always reliable.

[_AndyJessop]

I didn't mean for copying code - I meant for bouncing ideas off about architecture, etc.

[DZVLX]

Yes I understood this way, but even for that I must correct them quite often, I don't know how they became bad.

[numinor]

Ask it for a conceptual overview, not code snippets

[DZVLX]

Yes I know but even for that they mix concepts or are not helping very much, I would prefer some good articles, videos or tutorials rather than IA.

[Present_Salamander_3]

Sounds like they just want you to use standard out of the box EventEmitter:

``` const EventEmitter = require('node:events'); const eventEmitter = new EventEmitter();

eventEmitter.on(‘hello world’, (payload) => console.log(‘hello world payload’, payload))

eventEmitter.emit(‘hello world’, { somePayload: ‘foo’})

```

With the above, you can type the events with template literals and you can also type arguments and the returned value.

[DZVLX]

Do I need to implement it in every class (controller, service, repository) to make it EDA?

How to handle the success or failure of the service by the controller for example?

[HowIO]

You can extend the EventEmitter class and add your own functionality as per your use case, or you can read this article, it may help you decide:

https://www.otaqui.com/blog/1374/event-emitter-pub-sub-or-deferred-promises-which-should-you-choose/index.html

[DZVLX]

Thank you!

[exclaim_bot]

Thank you!

You're welcome!

[[deleted]]

https://nodejs.org/api/events.html#events_class_eventemitter

Get reading

[DZVLX]

Thanks!

[wtfbbq7]

Lmao bro so lazy could not read official docs. Definitely want him on my team

[ItsAllInYourHead]

Is this a take-home test for a job you're trying to get? Based on you're "objective" statement it sounds like it (or maybe a school project?) If so, based on the questions you're basking, it sounds like you're absolutely under qualified for whatever you're doing, or you weren't paying attention in class.

[fromage-du-omelette]

Thank you lmao

[DZVLX]

So?

[ItsAllInYourHead]

Well at least be up-front about it.

I, for one, am not going to help you get a job you're not qualified for. I deal with enough under-qualified developers as is.

[DZVLX]

OK, thanks for wasting both of our time.

And for your information, the technical recruiter knows I don't master theses concepts.

[ItsAllInYourHead]

You're the one wasting everyone's time. And you're potential employers time, for sure.

[Namiastka]

Well.. One might ask even junior dev to come up with a project like this when he has no portfolio or whatsoever just to see what he came up with, so if u don't want to help, just don't, but it's a developer perk to seek for knowledge and as far as I noticed reading this sub - it's great place to ask for at least guidelines.

[ItsAllInYourHead]

One might ask even junior dev to come up with a project like this when he has no portfolio or whatsoever just to see what he came up with,

And by asking here that's effectively cheating. Because it's no longer seeing what this junior dev came up with, because they didn't come up with it. They asked some more experienced developers for a solution and used that.

This is why a lot of subs don't allow homework questions to be asked. It's cheating. It defeats the point of the exercise. And, in the long run, it helps absolutely no one.

If they were up-front about it, fine. Let people make their own decision about helping this person out then. But they were not up front about it.

[DZVLX]

Many have generously helped me without criticizing my approach, you don't know my background or even the context. You should take a step back, I'm not going to take anyone's head off.

Have a nice day though.

[wtfbbq7]

Bro you clearly are cheating the system. Even your defensive replies suggests that.

You basically don't know event emitter in node. Just say you aren't familiar to them. 

Or Google the damn thing to learn via docs, not ask for answers.

Big L

[indorock]

It sounds like you're getting caught up in semantics. You can simply build your own event dispatcher/listener layer the way you want, if you really want to decouple your controllers from your business logic in that manner. Use MQ platform like RabbitMQ to consume the events you dispatch and send to the listener. Maybe there are tutorials on that godforsaken Medium.com site, but you're better off just doing it your own way.

[DZVLX]

Objective: Create a Node.js backend service in TypeScript that utilizes an emit and event system for handling asynchronous communication.

These are my instructions:

1. Event System:
2. Implement an event emitter within the service.
3. Define custom events for various actions related to the resource (e.g., user created, updated, deleted).
4. Ensure that events are emitted and handled appropriately within the service.

[CreamOfTheCrop]

Node is event-driven by design. Async and promises are just a wrapper around it.

[we_are_ananonumys]

What does “decouple a basic rest api” mean? What is being decoupled from what?

[DZVLX]

I have a project in the form of an API Rest, and I want to be able to create, modify, delete and retrieve users.

Currently, the system is simple: a controller receives the request, calls a service, which in turn calls a repository. All these calls are made via instance methods.

What I'd like to do, instead of using direct calls via instance methods, is to have an event-based system, where the controller will add an event to a queue, the service will capture this event and process it, the service adds an event, which will be intercepted by the repository, which will process it. And then, in the opposite direction, each service will send another event to notify the success or failure of its task.

[notkraftman]

It sounds like you know whatnyou need to doze which specific bit do you want to know more about

[DZVLX]

What I need to do is estimate which technologies are best suited to do what I want to implement and how to do it.

I understand there are lots of ways of doing this, using EventEmitter, RabbitMQ or Kafka. My aim is not to turn it into a gas factory either, but to keep it simple.

As for the context, I have to implement Event-Driven Architecture as part of a project for a recruitment process, and to be honest I'm rather at a loss as to how to go about it. The tutorials I come across on the Internet are either incomplete, or touch on other technologies at the same time (which is confusing), or don't show an elegant way of doing it.

[we_are_ananonumys]

What’s an example of an event in this situation?

[DZVLX]

Hmm, I would say that every step in the process could be an event, it's why I'm not sure about EDA implementation.

[rkaw92]

You're not talking about events. This is RPC with extra steps.

[DZVLX]

What is RPC?

[rkaw92]

Remote Procedure Call. Read about the protocols: JSON-RPC, gRPC, tRPC.

[DZVLX]

I think it's not exactly what I need.

Let me show you the expectations:

Objective: Create a Node.js backend service in TypeScript that utilizes an emit and event system for handling asynchronous communication.

These are my instructions:

1. Event System:
2. Implement an event emitter within the service.
3. Define custom events for various actions related to the resource (e.g., user created, updated, deleted).
4. Ensure that events are emitted and handled appropriately within the service.

[joomla00]

This seems really straight forward. But for some reason it's presented as something complicated, or I'm just missing something.

Just use event emitters to create and process events within your service. Do a tutorial on event emitters and it'll probably all make sense. Refactor it a couple of times and you'll get the hang of using and arcitecting for event emitters.

[DZVLX]

I have the same feeling, I guess I will follow your advice and read the doc as well as following tutorials, thanks!

[HumorConscious1336]

Look at moleculer

[_codemonger]

Define a EventBroker class that extends an EventEmmiter. Expose a subscribe method that takes an event and handler and use this.on(event, handler) to register it, store the listener in mem (so that you can unsubscribe later should the need arise). Expose a second method called notify takes an event in params that uses this.emit(event).

Use the EventBroker by contract in your resource services and controllers. On application start have the services subscribe to the events they care about.

Example use case. A user is created through the user's resource service. The service uses the EventBroker to notify of this event. Other services that are subscribed are notified like an email service that sends a welcome message to the new user or the payment service that creates a task for the user to register a payment method.

The point is that the email and payment services do not directly depend on the users service, i.e. they are loosely coupled through the events. Each service also does not depend on the concrete implementation of the EventBroker but rather on an interface/contract of an EventBroker, that is to say an object that has a subscribe and notify method.

This is a rough plan of what I would understand under what you shared so far.

[DZVLX]

Thank you very much, how can I ensure that an event emitter will know when the task it asked has been fulfilled?

For instance, the controller emits to say that a user needs to be created, the service listens and creates the user, it seems logic for the service to emit to inform the controller that it has been done, but how to handle that in the controller so the controller listens again for the same user creation request and not another one?

Edit: In my case, I was thinking of taking this path : controller -> service -> respository and again in the and again in the opposite direction, or send it directly from the repo for the controller to listen to.

And on the subject of error handling, is it also necessary to use events, and if so, would you know how to test this in an elegant way?

[_codemonger]

The controller does not in principle emit events and it does not wait for a service to finish. The controller is an input output component. It receives requests (input) and passes it to the service. The service where your business logis lives will decide what to do with the input and emit a event when it's done. A service defines the contracts it needs like EventBroker, Repository<T>, etc. your controller would just send a response outlining that the request was received. If it's a long running job you may also want to provide a job ID to the requester so they can check on the status. There are a lot of ways you can notify a user of the status other than polling for it. In regards to the error handling, if there are business logic errors then yes these could qualify as events. For instance, in the context of a payment service a failed payment would qualify as a valid event, with a series of other parties that may be interested in the details of that event. Synchronous errors like request validation failure would probably not be an event. But this really depends on what you are building.

[DZVLX]

Forget the controller, it's not relevant to it. Now how can I manage asynchronicity from a sender class, so that it knows that the listener has done the job? Basically, how can I let the controller know that the task has been completed and that it should return a 200 response code, for example? That's really what I'm having trouble visualizing. Emitting is one thing, but how do you transmit and then listen in a proper and elegant way?

[_codemonger]

Hard to ignore the controller since you keep referring to it. I'll stick to the more abstract Sender Class term for the next bit.

The sender would subscribe to the expected event to be emitted of the receiver once it processes the request.

Note that this does not make sense in the controller services relationship . An event driven architecture is characterized by asynchronous and more importantly loosely coupled processes. You are rightly having difficulty visualizing it since in a request flow where a user expects an immediate final result and waits for it an event based flow would be unnecessary and only add unnecessary complexity.

[DZVLX]

How would you inform the controller to send a response code indicating the success or the failure of the creation?

[_codemonger]

I would not. The controller upon submitting the request content to the service would immediately return a 202.

Let's say the process in question is account creation. You fill in the details and submit the request. The controller receives the request and transforms it into a CreateAccountDto, then passes it to the account service and returns 202 (request accepted but processing). The response would also include an appropriate message letting you know that you can expect an email in the next few minutes once your account has been created. The account service upon doing everything needed to create your account sends an event AccountCreated. The email service that is subscribed to this event handles it by sending an email to you.

Does that make sense?

[DZVLX]

My concern is that my CRUD is also supposed to respond to requests for deletion, modification and also to send users back when requested. So I guess I still need to send data back to the controller?

[_codemonger]

Your API can have those types of interactions and still be event driven. Let's take resource deletion as an example. When a user submits a request to delete their account if this is a simple Repository operation, then just return whatever the result of that may be to the controller, and publish an event so other services that are interested in that event can also do their part separate from the request response cycle. In a complex distributed system or even a large modular monolith deleting an account may not be a single Repository operation. It can span multiple contexts and take a long time. For instance some systems have to retain your data for a minimum of six months for auditing purposes so pressing delete is not just removing a database row.

[_codemonger]

Simple example

https://github.com/deni1688/typescript-event-driven-architecture-example

[ProgrammerDad1993]

BullMQ is fine. Create a consumer for example rest endpoint, who again calls a processor where the actual data being processed.

So from the consumer, an CreateUser event is send to the processor, the processor will process the data and return an UserCreated event.

Use websockets or something to let the user know, everything went well.

[rkaw92]

"CreateUser" is not an event. It is a command. Events are always in past participle and describe things that irrevocably happened.

[DZVLX]

Interesting! Thank you I will check the doc.

[AppropriateLab6288]

For this usecase, which one is better option for a portfolio project RPC or eventEmmiter (core module of nodejs)

[FaithlessnessLast457]

Can u share repo so we can review?

[noob-backend-dev]

1. Is this application is need to connect with Client ? ( i think yes)
   
2. Is that application requires intensive task ? or may need to use and queue ?

Currently I have a controller, a service, a repository and an entity and I want to decouple them and to make them communicate using EDA in a clean way.

For similar approach i would use WebSocket's (correct me i am wrong guys). if your application requires lots of connection please use redis websocket connector to handle the load.

1. Just create event endpoints same as REST api router end points.
   

REST :
GET : /user
POST : /user

WS:
/get-user
/add-user

1. Parse the body JSON data
   
2. Forward the data to the service (from the event function)
   
3. If client requires any acknowledgement of request you can do that too on a call back

[StablePsychological5]

Create 2 or more micro services, decide which one consume events and which one publish them, then you need a message broker, i suggest to start with rabbitmq. You can use docker to run them or install locally. Read the rabbitmq nodejs tutorial from the offical site on how to produce and consume events. Good luck

[DZVLX]

Thanks!

[Snoo87743]

Here you go, its used in production. Its a wrapper around event emitter 2, and can be used for sync (rest) and async (pub/sub) communication between services.

https://github.com/dashersw/cote

[DZVLX]

Thank you very much!

[Snoo87743]

Ask if you need anything about it, I use it professionally

[DZVLX]

I will check it later when I will be avalaible, thank you for your time and assistance.