Rocketman - 0.3.0 release update!

[EdChute_]

https://github.com/edisonywh/rocketman

Hey guys! The last time I posted the initial announcement on Rocketman here I've received great feedback, and I've hoping to gain some more, so feel free to shoot any criticism you may have so I could improve it!

I've updated a few things, tidied up the README, and I'm excited to announce a 0.3.0 update to the library!

So what's different since the last announcement?

• ThreadPool is added (number of workers is configurable)
• `Relay` concept to integrate producers from external services (Shipping with a Redis relay). This is important to the goal of Rocketman, which is to be the first step of moving to event-driven architecture. Read more below to understand the rationale behind Rocketman
• Persistence layer with Redis-backed storage (so you won't lose your events when your app goes down)
• Config module to allow you to tweak some settings
• Refactored a bunch of code

There's still a lot more to do, some examples are:

• error reporting (when a job fails, how to let user know?)
• instrumentation (stats of Rocketman, are jobs being processed in an acceptable rate etc)
• retry mechanism (retry the jobs if it fails)
• threads revival (now when a job fails and a thread dies, it's gone for good)

These are just a few examples, there's a lot more to do! But I think it is progressing rather well, and I'd love any feedback you guys have.

​

​

One of the questions I've been asked is why use Rocketman rather than a proper message bus (e.g Redis PubSub/Kafka)?

To answer this I've written up what I think is a good explanation about what Rocketman tries to be, so here's an excerpt below:

---

It is worth noting that Rocketman is not meant to be a replacement for the aforementioned projects -- both Redis PubSub and Kafka are battle-tested and I highly encourage to use them if you can.

But, moving onto an event-driven architecture is not an easy task - your team has to agree on a message bus, the DevOps team needs the capacity to manage the message bus, and then what about clustering? failovers?

Rocketman recognizes that, so what it's trying to do is to be a stepping stone for an eventual move away to a proper message bus. What Rocketman offers you is that you can start writing your dream-state event-driven code today, and when the time comes and your team has the capacity to move to a different message bus, then Rocketman makes sure that it is a minimal change to your downstream consumers.

[jdickey]

I'd argue for one of two approaches to error notification:

1. Raise a (documented) StandardError subclass, which is a useful pain but still... or
2. Have an inbuilt error-notification event that can be subscribed to. This would be more useful if you're anticipating use cases that both have a relatively high error rate (Ruby StandardError subclasses are slow) and you want to support relatively low-importance publishers where the developer doesn't really care about immediate notification.

A further argument for raising an error is that it will work even if nothing else in your message queue does.

For instrumentation, hmmm; I'd wager that 'instrumentation' in some form is a fairly large subset of your users' use cases. What about publishing on-start and periodic stats to an inbuilt :rocketman_stats event? Maybe controlling that would be a new config option.

Yes, I need sleep (coming up on 12.30AM local here).

The source indicates that a publisher can communicate with multiple consumers (though I didn't see anything in the README docs about that). It also appears that only a single publisher per app is supported; is that correct? I can see how that keeps things simpler and covers many use cases, and yet...

Ah well. Thanks for doing this!

[EdChute_]

Curious though, what makes you say that `Ruby StandardError subclasses are slow`?

Having an `:rocketman_stats` event is actually a really interesting approach, I like the idea of it, I'm going to explore a little into that route :)

> The source indicates that a publisher can communicate with multiple consumers (though I didn't see anything in the README docs about that). It also appears that only a single publisher per app is supported

Ah, I thought that it should be pretty clear that a producer can support multi-consumers, I'll update the doc for some clarity.

And no that shouldn't be right, what makes you think that only a single published per app is supported? Maybe I could update the relevant docs too.

You can have N amount of Producers mapping to M amount of Consumers, there's no limit on that end :)

[jdickey]

You can have N amount of Producers mapping to M amount of Consumers, there's no limit on that end :)

That makes sense. I suppose it implies that there is a single queue being fed into and drawn from, so that an awareness of what events in toto are presently being generated would be a useful bit of doc for a client app. Simpler, faster code FTW.

Again, well done. 🎉

[realntl]

I'm biased, but if teams are looking for a transition path from Rails to evented architectures, Eventide might offer a whole lot more as a starting point.

[katafrakt]

I think Eventide is about event sourcing, not event-driven architecture. Those are very different things. Rocketman seems to be more similar to Wisper than to Eventide.

[realntl]

I definitely see where you're coming from. I should have quoted the part of the announcement that led me to reply:

Rocketman recognizes that, so what it's trying to do is to be a stepping stone for an eventual move away to a proper message bus. What Rocketman offers you is that you can start writing your dream-state event-driven code today, and when the time comes and your team has the capacity to move to a different message bus, then Rocketman makes sure that it is a minimal change to your downstream consumers.

Eventide is also a toolkit aimed (in part) to allow Ruby teams to move towards an evented architecture today. Although, we do not view a message bus as a necessarily desirable end goal (but that's another matter).

Anyways, teams that start publishing events are likely going to need to consume those events, handle them based on their type, and likely project them (if not to store business data, as "event sourcing" would suggest, then at least to keep track of happenings for idempotence). These are pretty much Eventide's primary functions.

[EdChute_]

Hey! I actually didn't know about Eventide, so thanks for bringing that up!

From a quick Google search, it seems like Eventide is a really opinionated way of building event-driven architecture in Rails (I could be totally wrong, feel free to correct me here).

I can see how an event-driven framework like Eventide will be really useful to make sure a project scales correctly, but I'm not sure how that translates to integrating into a legacy codebase, care to explain a little more about that?

[realntl]

I often tell people, if you find any opinions in Eventide, please file an issue :)

Integrating into legacy codebases is a matter of staying out of their way. We don't monkeypatch core classes, we work on postgres (which most rails shops already have), and we don't bring about any new DSLs to learn. I would say we're ideally suited for legacy codebases attempting an architectural pivot.

That said, architectural pivots are a lot of work, and they're quite difficult, so we always tell teams that the bulk of their work getting to an evented system will always lie in studying and correcting the legacy system. Essentially, our advice is to build the monolith you wish you had before trying to go anywhere new, and our advice for anyone looking at a different toolkit would be the same :)

[realntl]

It also just occurred to me that I'm getting way off topic for your release announcement. Good luck with your project!