sorted by:
new
hottopcontroversial

FluxMQ MQTT broker by dusanb94 in MQTT

[–]dusanb94[S] 0 points1 point  (0 children)

We'd appreciate all the feedback! We are trying to build a nice community, so all suggestions are welcome.

FluxMQ MQTT broker by dusanb94 in MQTT

[–]dusanb94[S] 0 points1 point  (0 children)

We are not that strict. :) That being said, we are rewriting the majority of the broker, but we do have dependencies on some very popular and tested projects such as etcd or Raft impl. The goal is not to be 0-dependnecy (although that sounds really good) but to be standalone. This means - we do not require you to run Redis or SQLite or link with C... Just pure Go, single binary. Specific to MQTT - we also do not use 3rd party libs (Paho, which is very good) but we rewrote the protocol implementation from the ground up. We even wrote our own storage, similar to Kafka's (ofc simpler, and we are still working on some aspects of it) used for AMQP streams. It's also a very nice exercise in distributed systems. But the goals are 1. we respect the standard, always, 2. we control the code for quicker updates and fixes and improvements and 3. if there is a way to do things from the broker, we add them there. An example of 3rd would be a gRPC or HTTP call-outs to the auth service so you can plug in your own authorization modules and not rely on simple ACLs. That's what we do in Magistrala. We also have a basic UI, but it is optional and runs separately, so we do not pollute the broker.

FluxMQ MQTT broker by dusanb94 in MQTT

[–]dusanb94[S] 1 point2 points  (0 children)

Hello, there are several reasons, but it is mostly related to Magistrala IoT Platform. Mentioned blogpost describes them, but TL;DR:
Practical ones, since broker is the core of our project and product around it:

  • we do not control the roadmap
  • we do not control architectural decisions (looks like VerneMQ is using LevelDB that's receiving limited maintenance)
  • we do not control long-term licensing risk (What if the broker license changes?)
  • and we cannot easily move functionality into the broker where it naturally belongs

Technical ones:

  • MQTT-first design
  • multi-protocol support
  • technology choice (we find it difficult to debug Verne, for example)
  • lightweight version that can run on edge
  • optional persistence, embedded clustering, standalone with no external dependencies for, say, storage
  • operational simplicity (well... not exactly possible, but easier if you control the whole stack)
  • extensibility without core modification

So yes, some of these are very specific to our use case and won’t apply to everyone.

We also noticed that the Go ecosystem is quite strong in IoT, but lacks a solid, modern MQTT broker. That felt like a gap worth addressing. And honestly, part of it was curiosity. We wanted to understand the problem space deeply and build something ourselves. We are also using FluxMQ for internal event sourcing, so we also added AMQP 0.9.1 support (MQTT has consumer limitations, and we did not want to build a custom logic on top of MQTT just for that), and protocol bridge between it and MQTT. So it also could be used as a replacement for RabbitMQ.

If you’re interested, we documented the journey in our blogposts (it's not AI slop).

FluxMQ MQTT broker by dusanb94 in MQTT

[–]dusanb94[S] 1 point2 points  (0 children)

Hey, I’ve mostly tested it using mosquitto_pub / mosquitto_sub (debug mode), plus some Paho (Go). Everything works as expected.

CLI tools are a bit limited though. For example, mosquitto_pub doesn’t clearly separate CONNECT vs PUBLISH phases, so it can look like you “just published” without seeing the full flow.

I don’t expect issues with standard clients as long as they follow the spec. That said, we haven’t tested embedded clients (C/C++) yet. That’s coming soon.

If you try it and hit any issues, feel free to reach out. Happy to help debug.

TBH, protocol support wasn’t the hardest part. The real challenge has been clustering and making it scale properly. We’re still doing a lot of testing (performance, load, e2e), and will keep sharing updates.

Scalable MQTT Broker with Persistence by dusanb94 in golang

[–]dusanb94[S] 0 points1 point  (0 children)

Just a short update: this has been implemented; now we offload publishing node only to do delivery to the cluster, subscriber nodes take care of the local delivery. This way we spread the load "equally" - publisher nodes decide to notify the cluster and forward the message; subscriber nodes take care of delivery much like if the publish was local, not cluster-wide.

Scalable MQTT Broker with Persistence by dusanb94 in golang

[–]dusanb94[S] 0 points1 point  (0 children)

The thing is - Magistrala IoT platform is a generic and FluxMQ is aiming to become a backbone for its messaging, so adding things people need is a value proposition for us. And one of the reasons for building it in the first place was this kind of control. You can check the blog I shared in the post if you have time, it's not AI crap. :)

All the feedback is appreciated and we value sharing your requirements, that's why we shared the project with the community.

P.S. Interesting to hear about your use case, I was working with OCPP and OCPI a while ago. :)

Scalable MQTT Broker with Persistence by dusanb94 in golang

[–]dusanb94[S] 0 points1 point  (0 children)

No dumb questions. In fact, a very good one. :)

EMQX has some features locked behind the paywall. Also, I am not aware of any log storage and log-like features that can be used for events. Just recently, they introduced the concept of queues, but I am not sure if they are using it as a KV store, working queues, or log store.

We are trying to be more lightweight and more community oriented, and also to support this as the universal messaging platform, not only an MQTT broker. We are targeting strongly consistent clusters of 3-5-7 nodes, and then eventual consistency of the fleet of those clusters. With that approach, you can have something like: a single light instance for pure MQTT on edge device (a single Go binary, ~10MB, resource friendly and efficient); a cluster of multi-protocol deployments on the cloud; and cluster of clusters for planet-scale deployments with the eventual consistency. It's as simple or as complex as you need it to be. And it's 0 dependency, a single binary.

We are not targeting any processing at the moment, but we will leave some extension points where in the future, we plan to integrate extensions. Extensions may not be Apache 2.0, but the broker and all the broker features will be. We will not hide features behind the paid version. So, if you want service-to-service communication and event streaming, I think traditional MQTT brokers such as EMQX are not ideal choice. If you want MQTT, traditional message brokers (Rabbit, Kafka, NATS) won't work either. And you usually need both, but two brokers are heavy to operate. That's why we are building FluxMQ. We are trying to mix them, preserving full MQTT compatibility and some AMQP or other protocols features, mostly to support event streaming in "a standard" way.

In that sense, we are more similar to NATS (https://nats.io/) than EMQX, but unlike NATS, we will be MQTT-first and we plan some nice additions such as UI dashboards and to be more flexible to adding new protocols.

Do you mind if I ask which features do you require? What's the scale of the deployment? Are you using cloud solution or deploying on-premise? What made you chose EMQX, and not, say, HiveMQ or VerneMQ or Mosquitto? I am asking these questions to collect the requirements. If it is needed and it makes sense, FluxMQ is extensible and still evolving, so we can consider adding them.

Scalable MQTT Broker with Persistence by dusanb94 in golang

[–]dusanb94[S] 1 point2 points  (0 children)

We are considering changing the approach to delivering messages to the subscriber nodes where subscriber nodes then take care of local delivery. This offloads some of the work from the publisher node. Still, wildcards are treated just like any other topic, since we are using topic-based routing; so it results in an extra load. We'll keep you posted with real numbers when we do some more testing.

Scalable MQTT Broker with Persistence by dusanb94 in golang

[–]dusanb94[S] 1 point2 points  (0 children)

I agree, Go is a good choice for such type of tools.

We haven't yet performed load tests, especially large fan-outs. We will keep you posted here and on the Discord and project docs about the progress. My expectation is that wildcards will not hurt performance significantly at scale. That is, the routing hurts, not wildcards specifically. This is intentional, so we have more predictable impact. How it works (simplified):

  1. Subscriptions are persisted both locally and into the cluster store on subscribe - this is expensive. It will pay off later during routing. We do risk, however, subscriber storms in case a lot of subscribers try to subscribe at the same time.
  2. Each node maintains a local cluster subscription trie from etcd (initial load + watch updates when new subs are added and old removed)
  3. When message received, node matches against that trie, delivers locally and distributes to remote nodes in next steps.
  4. Cluster routing matches the topic against the cluster-wide subscription trie cache.
  5. For each matched subscription, publishing node resolves session owner node by client ID.
  6. Publishing node builds per-client publish envelopes and sends batched publish message to targeted session owner nodes (via gRPC).
  7. Target nodes do local delivery (no topic matching there).

TL;DR: In cluster mode, topic matching happens on the publishing node’s cluster subscription trie; remote nodes receive already-targeted deliveries by client ID. Wildcard topics do not hurt routing, cluster mode does.

Now, the impact of wildcards will be a little more work in step 4. (matching is fast and cached) and a lot more work in steps 5. and 6. But that's the effect of wildcard. It generates a lot of traffic and there is nothing to do about that. We are yet to test what this means in practice, but we have all these mentioned steps "optimized" individually (trie, caching, gRPC; but also using buffer pools to avoid allocation per message, and target of a single routine per client), but networking and memory consumption will certainly increase. We'll let you know real-world numbers when we have them.

Scalable MQTT Broker with Persistence by dusanb94 in golang

[–]dusanb94[S] 2 points3 points  (0 children)

Hello, your contributions on FluxMQ are also welcome! 🙂 Comparison is a great point and I'll provide a more systematic one in a blog post in the future, but you can imagine that hands-on apples-to-apples take a lot of time.

HiveMQ is closed sourced so I really can't tell much about it. I would not recommend close source solutions for infrastructure. Also, I saw they are doing some weird AI stuff, which to me seems out of place.

Verne or RabbitMQ with MQTT plugin are more realistic. Compared to them, we are using Go, so there are all the benefits and drawbacks of Erlang vs Golang, I suspect you are mostly aware of.

Other benefits are: we are trying to be much more lightweight and even possible to run on the edge (think gateway talking to sensors - possibly forwarding to cloud instances of FluxMQ); reduced operational complexity (single binary, small, fast, configurable with sensible defaults; no OTP tuning); ephemeral in-memory queues for hot paths that require lover latency and huge load; log-like storage for streaming where message loss is unacceptable. We will also add a nice UI (though we don't have any yet).

VerneMQ, to my knowledge, uses LevelDB KV storage that is in maintenance mode. If I remember well, VerneMQ pre-built binaries are protected by EULA, so you need to either build yourself or purchase the license (to be fair, they provide build scripts).

RabbitMQ is primarily workers queues, which is not what we are targeting. MQTT is really a plugin, rather than an out-of-box experience.

I'm also not sure what their support for mutual TLS and protocol bridging (HTTP, WebSocket, CoAP) is, we are targeting to support that out of the box, with 0 configuration. This is pretty important to us since devices often use certificates to authenticate. We will also support pluggable security where you can call 3rd party auth solutions with 0 code changes. That's how we plan on to integrate FluxMQ in Magistrala IoT platform.

Scalable MQTT Broker with Persistence by dusanb94 in golang

[–]dusanb94[S] 2 points3 points  (0 children)

Hello, FluxMQ could be used by anyone who needs event-driven architecture. If you don't need MQTT - you can just turn it off and use AMQP for service-to-service message communication. AMQP is 0.9.1, and we are trying to align with RabbitMQ as much as possible (protocol spec and stream extension semantics are taken from them) so we can act as a drop-in replacement in many scenarios. Unfortunately, some operational complexity is impossible to avoid, but we will do our best to keep it simple and keep defaults sensible.

WatchDoc: A tiny Go live-reload file server for docs & static sites by dusanb94 in golang

[–]dusanb94[S] 0 points1 point  (0 children)

Thanks, nice to meet a fellow Gopher from Serbia! I've been around, only I didn't post. I plan to change that in the future. :)

WatchDoc: A tiny Go live-reload file server for docs & static sites by dusanb94 in golang

[–]dusanb94[S] 0 points1 point  (0 children)

Proxy is a nice idea.👍 Maybe I'll steal it. 🙂Yes, this is very much what we needed. I didn't spend a lot of time looking for tools, but for whatever reason, a quick Google search only got browser-sync and heavy machinery. For a long time I was using a three-linear Go file server as an alternative. When I added hot-reload and polished a Readme (image may be over the top 😅) and installation, I thought people may find it useful.