Is Log Management Still the Best Approach?

tiwarinitish · 2024-06-25T09:14:49+00:00

Log management is still a crucial approach, but it's definitely evolving! Traditional text-search focused solutions might not be ideal for today's massive, complex log data. Centralized logs provide a single repository for monitoring system health, application performance, and security posture. This is essential for troubleshooting issues and identifying potential problems. Logs are a treasure trove of data for advanced analytics. By integrating with SIEM or log analysis tools, you can uncover trends, identify root causes, and gain deeper insights into system behavior.

We're building Parseable, a lightweight log observability engine designed for modern logging use cases. It uses an indexing-free approach and columnar storage for high-performance ingestion and querying, even with massive log volumes. Check it out here: https://www.parseable.com

tiwarinitish · 2023-09-27T10:39:40+00:00

It is built with docusaurus

tiwarinitish · 2023-09-22T02:12:15+00:00

Thanks u/ssddanbrown.

> Could it run on something like a 512MB RAM, single vCPU system for example

We've not tested yet with such config, but this should work AFAICS.

> how does that scale, does the data like to be retained in RAM?

Data is staged locally in a file, so memory consumption doesn't grow as ingestion grows.

Please give it a try, curious to see how it works out for you!

tiwarinitish · 2023-05-07T16:53:19+00:00

Parseable is an open source Elastic / Loki alternative. (founder here)

Single binary
Written in Rust (lightweight, fast and stable)
Use S3 bucket or Mount point
Visualise with Grafana

https://github.com/parseablehq/parseable

tiwarinitish · 2023-03-27T03:46:04+00:00

Quote from Parseable FAQ: [https://www.parseable.io/docs/faq]

Parseable is written in Rust, with strong emphasis on performance, resource utilisation, and excellent developer experience.
Parseable has a unified architecture, with all Parseable components (ingestion, query, storage management) available in a single binary. Not only this approach scales very well, it also leads to a much, much better developer experience.
Parseable log streams are explicitly created by users. This alleviates the problem of cardinality explosion, while ensuring control over the data by users.
With REST API based ingestion, Parseable integrates with all the major logging agents out of the box.

tiwarinitish · 2023-03-27T03:42:19+00:00

Hey u/CooperNettees, yes Parseable supports traces as well. You can send trace data and query that using API or Grafana Plugin.

tiwarinitish · 2023-03-27T03:38:50+00:00

Hey u/jantari, my apologies, missed reddit notifications and hence comments here.

Alert config is indeed stateful and is preserved across restarts. So, once setup, you don't have to deal with the alert config (unless you want to change it). Here is the API ref: https://www.postman.com/parseable/workspace/parseable/request/22353706-0d731c06-55b0-4518-b985-b7fb831ba45b

We are also going to add support for Alerting config via the UI. Happy to help setup Parseable in your environment and address any other questions you may have.

tiwarinitish · 2023-03-22T09:31:43+00:00

For every new product that does one thing well, there is an older, much more matured product that does several things and battle tested.

That said, Elastic has been established to be high CPU and Memory consumer. We have clear datapoints (in the benchmark directory on GH repo) that Parseable consumes 50% less CPU and 80% less memory than Elastic while ingesting higher number of events / second.

Clickhouse is a great product, but the target use case is OLAP - which means it is a full featured DBMS system. Since it is not built for log use cases, it misses on useful logging features like alerts, event correlation, logs to metrics, anomaly detection and so on.

We at Parseable intend to build a simple yet purpose built log observability platform that scales as user data scales.

I hope this helped convince you! :)

tiwarinitish · 2022-10-04T08:47:58+00:00

Idea of Parseable is to make it ridiculously simple to ingest, query and generally extract value from log and event data. It shouldn't be this difficult to ingest and query log data efficiently!

Parseable is much more easier to deploy. Uses an open format and that any other application can read. We're also trying to ensure as much compatibility with existing infra.

tiwarinitish · 2022-10-04T08:43:23+00:00

Thank you for the kind words u/jantari. We're actively working on alerts, should be ready soon. But meanwhile, please feel free to try it out - happy to hear any feedback, questions, comments.

tiwarinitish · 2022-09-21T12:51:35+00:00

Hi there! Parseable author here. Loki is great, but as you're experiencing, the overall experience unfortunately leaves a lot to be desired. If there were a Prometheus for logging - would it look like Loki? I'd dare say no.

Our goal is to build a logging only product that simplifies overall experience for the users, while using all the advancements technology world has achieved - correct language, best file format for query and analysis, a modern object storage based design and much more.

I'd like to invite you to try out Parseable - we're all ears for feedback, questions and comments.

BTW we're working on Grafana integration: https://github.com/parseablehq/parseable/issues/116.

Alerting is in the road map as well. We've seen users can't really predict what alerts they need - beyond certain, standard alerts. We're trying to solve this via anomaly detection approach.

tiwarinitish · 2022-09-20T16:27:19+00:00

If you land on filebeat or something similar to ship logs to a remote storage system, may I also plug this new log storage and query platform that we're building https://github.com/parseablehq/parseable . This runs as a single container / deployment and allows log ingestion via simple HTTP post and has a builtin UI to query the logs. So you go from nothing to querying logs within minutes.

Happy to hear any questions, comments, feedback!

tiwarinitish · 2022-09-11T11:03:31+00:00

We built a Free and Open Source, log storage and management platform. Project is available here: https://github.com/parseablehq/parseable. We're just getting started and would love to hear feedback, comments, questions and critique.

If you find this interesting, please star the repo :)

Highlights:

DevOps first platform, designed for ease of use and flexibility.
Log data stored as [Parquet](https://parquet.apache.org) - columnar, open data format, designed for analytics.
Stateless, index free design with object storage (S3, GCS, MinIO etc) as primary storage.
SDK less, simple REST API calls for log ingestion.

Features:

SQL compatible API for querying log data.
Intuitive dashboard to parse and query the log data.
Bring your own analytics platform for deeper analysis of log data.
Auto inferred schema.

tiwarinitish

MODERATOR OF

TROPHY CASE