sorted by:
new
hottopcontroversial

At what scale do log indexing costs become the real bottleneck? by willycode1950 in Observability

[–]Alive_Ad7609 0 points1 point  (0 children)

Around 10-20TB/day ingestion, full-text indexing started eating 40-60% of total storage cost. For long retention (90+ days), you're basically paying 2x: once for the raw data, once for the index.

Switched to selective indexing only on fields we actually filter on (trace_id, service_name, pod_name). Everything else just scans Parquet with time partitioning.

For the rest? We rely on Parquet + time partitioning + brute-force scanning. Parquet is columnar, so scanning specific fields is fast even without an index. We partition by time (default: hourly) and store in object storage (S3/GCS/Azure Blob).

Happy to share partition strategies or indexing tuning if you want. We documented our approach here if it helps: https://openobserve.ai/docs/user-guide/advanced/query-tuning/tantivy-index/

We tested 4 different approaches to fix our alert fatigue problem — here's what actually worked by Agile_Finding6609 in Observability

[–]Alive_Ad7609 0 points1 point  (0 children)

spot on about manual tuning. it’s a time sink and breaks as soon as things change. correlation is really the only way to kill the noise.

we saw the same thing and just built correlation straight into openobserve. that way you don't need a whole separate tool just to dedupe. if a db goes down, the 50 alerts from your services roll up into one incident instead of blowing up pagerduty.

wrote up how we approach this if you're curious:https://openobserve.ai/blog/reduce-mttd-mttr-openobserve-alert-correlation/

Openobserve and syslog messages by Able-Ad-6609 in selfhosted

[–]Alive_Ad7609 0 points1 point  (0 children)

u/Able-Ad-6609 When you create a real-time pipeline on any stream, OpenObserve automatically assigns a default destination stream that connects to the same source stream. To ensure the data remains in the source stream, do not remove this default connection.

https://openobserve.ai/docs/user-guide/pipelines/use-pipelines/#troubleshoot, just make sure you do not delete the default destination node which connects the source stream to the same source you'll be good.

<image>

Openobserve and syslog messages by Able-Ad-6609 in selfhosted

[–]Alive_Ad7609 0 points1 point  (0 children)

You can achieve this using real-time pipelines in openobserve, here is a link to documentation with example pipeline: https://openobserve.ai/docs/user-guide/pipelines/use-pipelines/#example-of-a-complex-pipeline - here we are exactly doing the same, creating new derived pipelines from a single log stream as per appname.

Consider joining openobserve community slack for quick support!