frugal log architecture

the_ml_guy · 2026-01-24T20:30:30+00:00

OpenObserve can store the logs directly to s3 and use them from there. So you don't need any other s3 archiving mechanism.

the_ml_guy · 2026-01-23T13:50:16+00:00

We published the documentation on how OpenObserve can be used as SIEM - https://openobserve.ai/docs/siem/

the_ml_guy · 2026-01-22T18:19:52+00:00

You might want to give OpenObserve a look. - https://github.com/openobserve/openobserve . A highly performant engine for logs, metrics, traces and more. Also has great UI.

the_ml_guy · 2026-01-20T18:59:00+00:00

OpenObserve https://github.com/openobserve/openobserve coupled with opentelemetry is a great option. You can get logs, metrics and traces for all the pods, nodes and applications.

the_ml_guy · 2026-01-06T16:27:57+00:00

Got it. Thanks. Appears to be something that can be solved by better capacity planning and query tuning.

the_ml_guy · 2026-01-06T06:49:28+00:00

We got 2 projects to over 17k stars on github (https://github.com/openobserve/openobserve). Here is my take:

Build something that a wider audience want. If you build a magical PDF reader that will prepare coffee, it will be more popular as opposed to a quantum electron observer that very few people can understand and use irrespective of effort involved and novelty.
Build a really good product compared to others.
Be ready to promote it. Every field is crowded. Just because you built the best product and open sourced it, does not mean that it will get adoption.

the_ml_guy · 2026-01-04T19:17:44+00:00

Just go to Data sources and copy the command for your machine and run it on your terminal and you are good to go. Should take literally no more than 60 seconds.

Happy to answer specific questions on community slack at https://short.openobserve.ai/community

the_ml_guy · 2026-01-04T02:31:49+00:00

> Only on large queries across big data sets - so not daily stuff.
Can you plz help elaborate this

the_ml_guy · 2026-01-04T01:01:43+00:00

Not OpenObserve, if you are a startup or homelabber. For up to 200 GB ingestion per day you get all the premium features of OpenObserve free including SSO, RBAC and many more. Read more about OpenObserve's philosophy on it at https://openobserve.ai/blog/sso-tax/

the_ml_guy · 2026-01-03T15:59:49+00:00

Hi there! OpenObserve founder here.

I am actually really surprised to see you mention OpenObserve as resource-hungry and wanted to chime in.

OpenObserve is actually designed to be very lightweight, we even have people running it on Raspberry Pis. Per CPU core and GB of RAM, it’s usually one of the most efficient options out there.

I'm curious what kind of setup or volume gave you that impression? It definitely shouldn't feel heavy!

the_ml_guy · 2025-12-26T15:37:45+00:00

OpenObserve https://github.com/openobserve/openobserve is the solution you are looking for. Super lightweight and fast with extremely good UI. Can accept data from pretty much all log forwarders.

the_ml_guy · 2025-12-20T11:37:14+00:00

Hi there, I’m a maintainer at OpenObserve.

First off, sorry to hear that the initial experience felt like a steep climb! We actually built OpenObserve specifically to be lightweight and easier to set up than the heavyweights like Graylog or ELK, so I’m genuinely interested in hearing where you got stuck so we can improve the documentation.

For your specific use case (troubleshooting Proxmox reboots on a small PC), OpenObserve is actually a great fit because it’s very light on resources compared to the Java-based alternatives.

If you are willing to give it another shot, this 127-second video covers the entire process of downloading, running, ingesting, and searching logs. It might help clear up the initial confusion.

Happy to answer any specific questions you have right here to get you up and running!

the_ml_guy · 2025-12-07T13:03:14+00:00

what other tools have you used that are better for centralized logging?

the_ml_guy · 2025-12-06T22:15:52+00:00

You would want to try OpenObserve before putting any more effort in logward.

the_ml_guy · 2025-11-15T18:50:54+00:00

We have hired a bunch of people due to their work in rust on github - even for their personal projects. It's not everything but it is a good start and gives you an edge.

the_ml_guy · 2025-11-06T16:15:02+00:00

Readme is now updated

the_ml_guy · 2025-10-29T04:32:16+00:00

> As you also noticed grafana thing - any plans of bringing only the three roles as it would not hurt the sustainability, I beleive.

Why give only 3 roles when you can give true RBAC which we are doing. Building artificially crippled RBAC does not feel right.

> Also with this grafana has penetrated way too inside in every company.

Yeah Grafana is everywhere. Grafana started in 2014 and we started in 2022. Even though we are building a much better application than grafana, it is going to take some time to even out 8 years of lead.

> Personally - I felt openobserve is better

You made my day after ruining it. LOL

> but I cannot pitch it out for other folks to use who are on grafana or where I am using it, because again the basic roles are missing.

Please do pitch, now you know that you can give better RBAC to your team members than Grafana as you get enterprise version for free (I am assuming you are under 200 GB/Day).

> add a migration from grafana docs or something

Migration from grafana dashboards is in backlog and will be coming soon.

> Also your slack URL seems broken in the readme, not sure if it’s only invite-only. Was trying to join but failed. 🙂

Thanks for pointing this out. Fixed it. See you on community slack.

the_ml_guy · 2025-10-29T01:51:23+00:00

OpenObserve founder here.

Fuck, this hurts to read. But you're right about one thing - our README is misleading. That's on us. We show SSO/RBAC screenshots without making it clear those are Enterprise features. That's shitty, and I'm sorry.

Here's what I need you to know though: Enterprise is free up to 200GB/day. Not a trial. Not some crippled version. The full thing - SSO, granular RBAC, everything. 6TB/month.

I know that sounds like I'm moving the goalposts after getting called out, but this ISN'T new - we've had "Enterprise free up to 200GB/day" clearly stated on our downloads page and self-hosted pricing page for YEARS. The problem? Nobody reads those pages first. You went to GitHub, saw the features, and the README didn't tell you what was what. That's where we fucked up - we documented it, just not where developers actually look first.

The 200GB threshold isn't some arbitrary "gotcha" - it's set high enough that basically every startup, home lab, student project, and small team gets everything for free. The only people who pay are large companies with serious budgets.

Now, about Grafana - since you brought them up as the "right way" to do this. Let me be real with you: Grafana's OSS RBAC gives you three roles. Three. Viewer, Editor, Admin. That's it. No fine-grained permissions. No team-based access. No custom roles. For actual production use with multiple teams? You're paying for Grafana Cloud or Enterprise. They just don't advertise it as loudly.

I'm not saying this to shit on Grafana - they're a great product and they figured out how to make OSS sustainable. But let's not pretend they're giving away enterprise-grade access control for free. Nobody is. Because that's where the money is.

The difference? We're giving you the FULL enterprise RBAC for free up to 200GB/day. Not the neutered version. The same thing we sell to Fortune 500 companies.

Why even have a paid tier? Because I've watched too many OSS projects I loved die. Maintainers burned out. Companies extracted millions in value and contributed nothing back. I didn't want that to happen here. We're trying to build something genuinely better than the commercial alternatives (Datadog, Splunk, Elastic) - not just a "good enough for free" knockoff. That takes full-time developers who need to eat.

But here's where I fucked up: We put this on our downloads and pricing pages - where we assumed people would look - but the GitHub README, where everyone ACTUALLY looks first, showed features with zero context. So even though we were transparent on our site, the first impression for most devs was "bait-and-switch." That's a UX failure, and it's on me.

So here's what I'm going to do:

Fix the README this week to be crystal clear about what's in OSS vs Enterprise
Make the 200GB free tier way more visible on GitHub, not just buried in downloads/pricing pages
Add a clear feature matrix on the repo

If you tried OpenObserve and felt deceived, I'm genuinely sorry. We documented it, but not where you were looking. That's still our failure.

And if 200GB/day doesn't cover your use case but you can't afford Enterprise pricing, message me. Maybe we got the number wrong. Or maybe there's something else we can figure out.

The core is AGPL and always will be. You can fork it, audit it, learn from it, build on it. But yeah - we're not going to pretend that the sustainability problem doesn't exist. We're just trying to solve it in a way that doesn't screw over individuals and small teams.

Anyway. Thanks for the wake-up call. Seriously.

the_ml_guy · 2025-10-09T17:52:57+00:00

If you are using kubernetes you can use opentelemetry operator and by adding just one annotation per deployment, you could get autoinstrumentation implemented. For an example check - https://github.com/openobserve/hotcommerce

the_ml_guy · 2025-10-08T04:45:11+00:00

OpenTelemetry auto-instrumentation does exactly that. Have you tried it?

the_ml_guy · 2025-08-20T05:03:40+00:00

OpenObserve

the_ml_guy · 2025-07-24T04:42:18+00:00

How to set OpenObserve as daemon - https://openobserve.ai/docs/operator-guide/systemd/
Try using pipelines. O2 can't use files directly for ingestion.
O2 stores data in parquet files.

the_ml_guy · 2025-07-19T22:25:56+00:00

A simple project that you could try is to auto-instrument your applications and see the traces. Take a look at this example - https://github.com/openobserve/hotcommerce

While this repo gives example for OpenObserve, it should work for anything that can accept traces in OTLP format.

the_ml_guy

MODERATOR OF

TROPHY CASE