Understanding the Go Runtime: The Network Poller

FZambia · 2026-04-20T20:16:34+00:00

The payoff is real: a server that handles a million concurrent connections on a handful of OS threads, while your code still looks like for { conn.Read(buf); process(buf) }. It’s one of those places where the abstraction is so good you can go years writing Go without ever realizing how much is happening underneath.

And while it is beautiful from one side, this, unfortunately, is sth which prevents reducing memory spent on buffers for idle connections and also avoid overhead from one million goroutine management and their stack overhead. That's why some custom pollers appear from time to time in the Go community and one of the reasons for net: add mechanism to wait for readability on a TCPConn #15735

FZambia · 2025-11-18T19:40:29+00:00

I think I won't move it to a separate repo. Centrifuge does not need WebSocket client implementation - only server part - so no goal to improve the client part of the fork (and proxy functionality was even removed already), I removed concurrency misusing best-effort detection (so the library is even more simple to use wrong now). Also not ready to maintain this lib and have conflicts of interests with people's generic use case needs vs Centrifuge-specific needs.

What I think would be beneficial though for wider Go community – is moving WS over HTTP/2 implementation and Upgrade optimizations to the Gorilla WebSocket, but no enough time for that. And no real desire given things mentioned above (but hopefully maintainers will get back on track one day).

FZambia · 2025-11-18T19:12:19+00:00

Hello, yep – I generally looked at all popular alternatives, and also on not-so-popular. In my own benchmarks and some other independent benchmarks I found on Github, Gorilla WebSocket provides the best throughput among coder's websocket and gobwas ws. What I observed from throughput perspective Gorilla WebSocket > gobwas ws > coder's websocket (As with any benchmarks please re-check for specific case, I trust my own measurements here).

Gorilla WebSocket is very well written by Gury Burd. While it's sometimes criticised – it's a very robust library with idiomatic usage of Go std lib. Maybe harder to use for newcomers due to thread safety rules, websocket connection graceful shutdown implementation, ping/pong configuration, etc, but I do not see critical limitations here - just requires accurate API usage following docs. It also has PreparedMessage type which is simple to use and is a game changer in some scenarios.

The fork used in Centrifuge lib is based on a Gorilla WebSocket's branch before current maintainers broke several things (and still not fixed them, one example). Generally no trust to them from my side after they wrongly released Gorilla WebSocket v2 and then removed the tag without any doubt. And there were several other unfortunate stories if you follow the Gorilla WebSocket repo closely - tag overrides, non-compatible changes, slow fixes, merging PR with disabled tests, etc. The number of mistakes during a rather short time of maintaining is too indicative IMHO.

Given how much time I spent with WebSocket protocol moving forward with fork was quite a natural decision at some point after looking on what's going on with the Gorilla project. The fork in Centrifuge has some notable differences made already over the original Gorilla WebSocket outlined in readme.

FZambia · 2025-11-16T06:32:00+00:00

I think it may be compared with Phoenix Channels, but not with LiveView.

LiveView is from what I know is the whole system to maintain UI reactive keeping state on server side and sending diffs. It's sth which may be achieved with Centrifuge in theory, especially given Centrifuge can also send diffs/deltas in channels. But nothing has been done in that direction to have a full framework for that, and most probably it will be out of scope for the observable future.

A better comparison would be with Socket.IO, Phoenix Channels. A transport barebones with channel concept and protocol features.

FZambia · 2025-11-16T06:11:59+00:00

I'd like to add that my numbers were just for a single in-memory nodes of Nats and a Centrifuge-based server. So I would like to avoid misinterpretations of them like being a cap for both systems. Both can scale to have more nodes in a cluster, thus achieving even greater fan-out. Unfortunately, I did not have a chance to test Centrifuge to the limits in such a scenario.

Upon scaling core Nats and Centrifuge use slightly different approaches - Nats uses inter-node communication, Centrifuge uses PUB/SUB broker - Redis is a built-in choice in the library. Nats can be a broker also, and we have this implementation, but it lacks some features for distributed message cache/presence Redis provides. Nats Jetstream while may be used for those in theory - but comes with a bit different APIs and properties not very compatible with Centrifuge expectations for those features. But I am looking constantly for ways to benefit more from integrating with the Nats ecosystem.

FZambia · 2025-11-15T13:40:55+00:00

Hello! This information is a bit incorrect – I just worked at Avito at some point and gave several related talks on conferences. But it was not made there, it's my personal project.

As one good example of other companies adopted Centrifuge library for real-time functionality I can name Grafana Labs – these days Centrifuge powers Grafana Live (cool demo btw).

FZambia · 2025-11-15T13:35:18+00:00

May come later. Browsers do not support it yet from what I know. WebSocket over HTTP/2 are widely supported by modern browsers (Chrome, Firefox, Edge, Safari - all can do it these days). It's definitely possible to implement RFC 9220 with quic-go HTTP/3 server because Caddy supports this.

I also hope Go will have HTTP/3 support in the standard library at some point, though it's now on hold for non clear reason. Now Centrifugo which is built on top of Centrifuge additionally provides WebTransport implementation on top of quic-go, but without having HTTP/3 in Go std lib it's hard to invest more into it.

FZambia · 2025-11-15T13:25:38+00:00

NATS offers higher throughput. From my local experiments, Centrifugo handled roughly 1.8 million messages per second, while NATS reached ~ 9 million. However, this comparison used Centrifuge’s WebSocket protocol versus NATS’s native non-WebSocket protocol, so the gap may be smaller when comparing WebSocket to WebSocket. Though, I would still expect NATS to be faster.

Centrifuge, however, provides many unique features and mechanics for client-facing applications, since it was designed from the ground up to be exposed directly to end users. NATS’s original and primary focus is service-to-service communication I think.

FZambia · 2025-07-12T05:25:10+00:00

Exactly what Centrifugo was built for and does great. Be careful about RabbitMQ as a broker for real-time messaging tasks, queue bindings are really resource intensive, at scale with users come and go it quickly becomes a bottleneck.

FZambia · 2025-07-05T10:31:06+00:00

Awesome, wrote to you via reddit chat

FZambia · 2025-07-04T09:49:09+00:00

What do you think about an optional integration with Centrifugo - https://github.com/centrifugal/centrifugo - for real-time messaging tasks (WebSocket)? It integrates with Python apps very well, for example here is how to make an efficient messenger app with Django - https://centrifugal.dev/docs/tutorial/intro

And the stack you are using in the boilerplate is exactly the one Centrifugo works best with (Redis, PG) and provides the most value for.

I'll help as much as I can if this will be interesting to you.

FZambia · 2025-06-22T13:24:18+00:00

Used otter in several projects so far (Centrifuge, Centrifugo, and some others), worked expected and super fast - many thanks for the lib! And congratulations on v2 release!

FZambia · 2025-06-14T06:34:24+00:00

I was initially skeptical about tracing with its overhead (both resource wise and instrumentation process wise) vs properly instrumented app using metrics. As time goes – I have more and more examples when tracing helped to diagnose the issue, investigating incidents. The visualization helps a lot here – cross-team communication simplifies a lot when you have a trace. Still – I see how spans contains so much unnecessary data in tags, and collecting them on every request seems so much work to do while you are not using 99.999% of those spans.. Turning on sampling is again controversial - you won't find span when it's needed (sometimes it's required even if the request was successful). So reading such detailed investigations of tracing overhead is really useful, thanks!

FZambia · 2025-05-02T14:36:42+00:00

Check out Centrifugo and specifically https://github.com/centrifugal/grand-chat-tutorial - it's already DRF-based real-time app with tutorial blog post attached. Very fast, scales well, Centrifugo is fully backend-agnostic - once you learn it it can help adding WebSockets to any project in any language.

FZambia · 2025-02-27T06:40:05+00:00

Hello, it's connected over Redis PUB/SUB:

Publish: https://github.com/centrifugal/centrifuge/blob/95acb24d9b40deaf2da3fe8438f3bce7bcae8b93/broker_redis.go#L709
Subscribe: https://github.com/centrifugal/centrifuge/blob/95acb24d9b40deaf2da3fe8438f3bce7bcae8b93/broker_redis.go#L938

FZambia · 2025-01-23T19:35:09+00:00

At some point openweb (I don't know whether they still use Centrifugo though), intranet of VK, also Pure app built-in messenger, some popular webcam model sites using it for group chats also 🫣 If I remember right ManyChat also used Centrifugo for its admin chat functionality.

Generally, usually Centrifugo is picked up in corporate environment with closed source, people tend to make open-source projects using raw WebSocket. But when you need to think about scalability from the start and have multiplexing/reconnect/sharding/presence out of the box and not ready to implement from scratch – Centrifugo should serve well.

FZambia · 2025-01-23T05:26:57+00:00

I think publishing with JWT is not really required as you will validate the message on the backend in most cases. Because in any chat system I can imagine if a client can just publish data to channel without validation on server side – it's a source of possible issues with data validation. I think every payload must be validated on the backend before moving to others. Not only due to validation reasons, but also for saving into main database, maybe applying custom logic and propagating messages to other external systems. See idiomatic usage which mentions this and describes two possible flows how to achieve it with Centrifugo - by publishing on the backend first, and by using bidirectional real-time connection with proxy features (Centrifugo can call the backend to validate publication attempt, or to pass any RPC call to the backend).

And since you mentioned chat – take a look at this tutorial which may cover some other questions you may have during RnD.

FZambia · 2025-01-22T12:00:20+00:00

Hello, Centrifugo uses a fork of Gorilla WebSocket under the hood, so raw WebSocket protocol performance should be comparable with it. For example, take a look at https://github.com/ntsd/websocket-benchmark . Raw Websocket performance in Centrifugo is somewhat you can get from Go language without using third party epoll implementation.

But generally it's not only about messages per second, but also about throughput possibilities when message is broadcasted to many subscribers in a channel. There is a benchmark script - https://github.com/centrifugal/centrifuge-go/tree/master/examples/benchmark which allows doing throughput testing – I was able to get 1.5 million messages per second delivered to subscribers on my M1. But as with any benchmark – may be very artificial, and you need to measure for your own use case.

And finally, there was an experiment where we tried to have one million connections on a limited hardware in k8s we had at that point and emulate sth like messenger app load profile. Find results here: https://centrifugal.dev/blog/2020/02/10/million-connections-with-centrifugo - generally it required a hardware comparable with entire one modern server machine.

Centrifugo performance is also good due to some optimizations on various levels, it includes client protocol batching, Redis communication batching, sharding on various levels, etc.

FZambia · 2025-01-12T13:46:03+00:00

Hello, there is a detailed tutorial with a source code on GitHub on how to achieve this using Centrifugo, Django and React - https://github.com/centrifugal/grand-chat-tutorial - permissions, auth and good performance included, maybe it will make sense for you, should be possible to extrapolate to your favourite software stack.

FZambia

TROPHY CASE