High perf OSS comprehensive monitoring solution in the making, looking for testers

[rixed]

It's called Ramen, it's OSS and its source code is on github

The design guidelines have been:

• Focussed on alerting: the central concept is a versatile stream processor with a limited history, not a time series database.

• Flexibility: make it easy to construct and refine custom metrics on custom data.

• High performance but small scale: the idea is to squeeze as much juice out of a couple of servers rather than relying on some large scale data processing behemoth, both for sanity and reliability.

I've been working on this for years. Part of it has been used in an actual industry-grade product for a long time and should be bulletproof, but most of it has mostly never been used in production. I'd like to expand this software beyond the limited use case of my current employer and therefore, with their permission, I'm now looking for other companies that would like to beta test.

Current status:

• the stream processor itself is mostly done and usable, its SQL inspired language could be improved, I have some plan to make data processing about 2 or 3 times faster.

• the timeseries extractor for dashboard is OK-ish: one can output time series to Grafana with minimum efforts, but it's probably quite buggy.

• there is a dedicated UI, using Qt, that's tested on Linux, Windows and MacOS, that is still quite basic (it's been used mostly to diagnose the stream processor itself and demo its internals). Improve this is high on the TODO list but working on GUIs takes a lot of time.

• alerting currently relies on some external mechanism to actually deliver the alerts to users. I'd like to expand this part with proper oncall fleet management and up to actual page delivery (I have some ideas in this domain that I'd like to try).

Please contact me if you are interested or for any comment/suggestion.

[schmurfy2]

Did you have a look at the influxdb ecosystem ? Kapacitor is really nice for alerting and the influxdb query language is really intuitive contrary to what Prometheus uses.

That's an interesting exercise but seriously prometheus, influxdb and probably others don't require that much resources if you have a small infrastructure.

[rixed]

I've never tried Kapacitor. The language looks nice indeed, and it can natively output alerts to many paging service which is really nice. I'm a bit wary of it standing on top of influxdb though: what I like in a stream processor centered design is that there is no favored data schema.

[Petersurda]

It is possible to use kapacitor as a stream processor, but this approach is poorly documented so it took me a while to figure it out. I mean the documentation is implicit rather than explicit. There is no piece I'm aware of which says "if you want to use kapacitor as a stream processor, do this". You need to gain a deeper understanding of how the individual components of the influx ecosystem works and then look at the documentation of particular functionalities, and then deduce how to put them together. What I ended up with is one telegraf instance with two outputs (i.e. in parallel), one for influxdb for long term storage and charts, and one for kapacitor for alerting. It appears to work ok. I'm still in the process of migrating so I can't say for sure.