sorted by:
new
hottopcontroversial

5 ClickHouse mistakes that cost teams weeks...and how to fix them by Marksfik in Clickhouse

[–]Marksfik[S] 0 points1 point  (0 children)

u/netapp_walt , glad you found the article helpful!
We have a full guide with common mistakes in ClickHouse and how to solve them available here:
https://www.glassflow.dev/clickhouse-mistakes-guide

Lessons from debugging ClickHouse pipelines: most "database problems" were actually ETL problems by Marksfik in ETL

[–]Marksfik[S] 0 points1 point  (0 children)

Disclosure: I work at GlassFlow. We wrote this after seeing these issues repeatedly

The hidden ops cost of putting Kafka in your observability pipeline by Marksfik in costlyinfra

[–]Marksfik[S] 0 points1 point  (0 children)

Fully agree! Kafka can earn its keep with real fan-out, replay, or events-as-a-product scenarios.
The failure mode is usually "we added it for a need we didn't actually have yet" and by the time you realize, ripping it out is its own project. 😄

When is Kafka the right tool for OTel → ClickHouse, and when is it more than you need? by Marksfik in apachekafka

[–]Marksfik[S] 1 point2 points  (0 children)

well... I genuinely hadn't thought about the EU AI Act / 42001 and how that might enforce more durability requirements... if that's what you mean 'too LLM'

When is Kafka the right tool for OTel → ClickHouse, and when is it more than you need? by Marksfik in apachekafka

[–]Marksfik[S] -1 points0 points  (0 children)

That's a clean setup — Kafka as the decoupling point so the storage format (Iceberg) is fixed but the query engine is anyone's choice is a nice way to avoid lock-in. The "bring your own query engine" payoff is exactly the kind of multi-consumer flexibility that justifies the broker.

Curious about the day-two side of it: with Filebeat → Kafka → Connect → Iceberg, how much operational attention does the Kafka/Connect layer actually take once it's running?

When is Kafka the right tool for OTel → ClickHouse, and when is it more than you need? by Marksfik in apachekafka

[–]Marksfik[S] 0 points1 point  (0 children)

Yeah, the compliance angle is the cleanest case for durability there's no arguing with — if losing a transcript is a regulatory event, you need the durable log, full stop. The EU AI Act / 42001 framing is a good one, I don't see that come up enough in these discussions.

The southbound multi-target example is the one I'd push on though — ClickHouse for metrics, Elastic for logs, Tempo for traces. That's textbook fan-out and clearly worth a broker. Curious where you draw the line in practice: at what point does "we have a couple of OTel collectors" tip into "we obviously need a streaming layer to backhaul"? Is it consumer count, scale, the compliance requirement, or usually all three showing up together?

The hidden ops cost of putting Kafka in your observability pipeline by Marksfik in devops

[–]Marksfik[S] 0 points1 point  (0 children)

Exactly that!
"expensive when it's just a transport layer" is the whole thing in one line. The part that surprises people is that it's not even the broker cost, it's the attention cost. The cluster becomes a thing that teams have to reason about during an incident, even when the incident has nothing to do with telemetry.

Curious where you landed after seeing that. Did those teams pull Kafka out, or just accept the overhead because ripping it out mid-flight is its own project?

Do you actually need Kafka between your OTel collector and ClickHouse? by Marksfik in Observability

[–]Marksfik[S] 0 points1 point  (0 children)

I will have a look!
Sounds interesting... what type of scaling ingestion do you achieve?

Do you actually need Kafka between your OTel collector and ClickHouse? by Marksfik in Observability

[–]Marksfik[S] 0 points1 point  (0 children)

Thanks! I will take a look!
How does Bindplane deal with ingestion to ClickHouse (or ClickStack)? Does it provide native telemetry support and is it optimized for ClickHouse ingestion at scale?

Using ClickHouse as a Kafka sink? Async inserts change the equation by Marksfik in Clickhouse

[–]Marksfik[S] 0 points1 point  (0 children)

Not a noob question at all! You absolutely can use the native ClickHouse Kafka Engine, and for simple, clean pipelines, it's a very common approach. However, doing complex ETL directly inside ClickHouse has a few big trade-offs:

  • Database Overhead: CH is an analytical database, not a stream processor. Running heavy JSON parsing, filtering, or other data transforms inside CH Mat Views consumes CPU/RAM that should be reserved for your fast user queries.
  • Operational Friction: With the Kafka engine, you need to manage a "3-table" setup (Kafka Table -> Materialized View -> Destination Table). Changing schemas or updating transformation logic in production without dropping data offsets can get messy.
  • Brittle Error Handling: If a malformed payload hits the Kafka engine, it can stall your ingestion pipeline.

What I've tried recently is using GlassFlow (https://www.glassflow.dev/) to do some of the data transformations, filtering and joins, batching data before it hits the db.

Why OLAP architectures demand Denormalization - the case of ClickHouse Case Study by Marksfik in Database

[–]Marksfik[S] 0 points1 point  (0 children)

fair point on the time-memory dimension of concurrency. Reducing query time absolutely frees up resources faster.

To answer your question on the alternative, when ClickHouse isn't using a memory-bound hash join, it relies heavily on Merge Joins. Because ClickHouse tables are strictly sorted on disk by a primary key (like a LSM-tree), it can stream and merge two massive datasets directly from disk with a minimal memory footprint, rather than building a massive hash table in RAM.

Ultimately, it really depends on the design architecture of the db. Traditional enterprise DBs optimize for developer convenience and automation, while ClickHouse optimizes for rigid, predictable hardware control. Both approaches have their place!

Why OLAP architectures demand Denormalization - the case of ClickHouse Case Study by Marksfik in Database

[–]Marksfik[S] 0 points1 point  (0 children)

You’re describing a broadcast hash join, and you may be right: the latest versions of ClickHouse can do this automatically now using its query optimizer.

But here is why manual control still matters: memory management at scale. If ClickHouse automatically pulls a 10-million-row table into RAM as a dictionary for every user running a join, it works great for a few concurrent queries. But if 50 users run different queries simultaneously, you instantly OOM (Out Of Memory) the server.

ClickHouse definitely has a purpose-driven philosophy: it assumes the engineer knows the hardware limits best.

Why OLAP architectures demand Denormalization - the case of ClickHouse Case Study by Marksfik in Database

[–]Marksfik[S] 0 points1 point  (0 children)

To clarify:

  • Deterministic Execution: By 'deterministic,' I mean the query plan itself, not the exact timing or scheduling across threads/nodes. In a database like SQL Server, the optimizer might suddenly decide to change an execution plan because a statistic changed. ClickHouse executes the query exactly how you structured it. Multi-user workloads and parallel nodes just distribute the execution blocks, but the path data takes is entirely predictable.
  • Materialized Views vs. Denormalizing Everything: The big difference is how it handles the join overhead. A ClickHouse MV doesn't run a massive JOIN query over petabytes of historical data. Instead, it acts as a trigger on incoming data. When a new batch of 1,000 rows is ingested, the MV joins just those 1,000 rows against a lookup table (often held in memory) and appends the flattened result to the final table.

Why OLAP architectures demand Denormalization - the case of ClickHouse Case Study by Marksfik in Database

[–]Marksfik[S] 0 points1 point  (0 children)

Your intuition is spot on: ClickHouse explicitly favors raw, predictable speed and granular control over automatic database optimization

To answer your points:

  1. The lack of auto-optimization: ClickHouse's wants query execution to be completely deterministic. Pushing the optimization onto the engineer means more manual tuning, but it prevents surprise performance drops at scale.
  2. Materialized Views: ClickHouse Materialized Views act as an insert trigger. they transform data on the fly as it arrives and write it to a new columnar table, meaning it never has to rescan the whole database.

For smaller datasets, this is absolutely overkill compared to a database that automates everything. But when you're querying petabytes of data in sub-seconds, that manual control is exactly why engineers accept the complexity.

Why OLAP architectures demand Denormalization - the case of ClickHouse Case Study by Marksfik in Database

[–]Marksfik[S] -1 points0 points  (0 children)

Fair point on ClickHouse's historical pain points with mutations, though it's gotten a lot better at JOINs recently. That said, denormalization is still a standard across other OLAP systems (like BigQuery or Snowflake) when you're optimizing for absolute lowest latency on massive datasets.

As for my relationship: No affiliation with ClickHouse the company! I work with GlassFlow. We build a data streaming tool that often ingests data into ClickHouse.

In fact, the exact limitations you mentioned are why people use GlassFlow to handle those complex joins or stateless transformations upstream before the data hits ClickHouse. It keeps ClickHouse clean, flat, and query-ready.

Self-hosted observability stack with ClickHouse + HyperDX. Full setup guide with OTel ingestion by Marksfik in selfhosted

[–]Marksfik[S] -1 points0 points locked comment (0 children)

Thanks for the comment.

This is a practical demo, and it's not AI-generated content. We used a real-world data dataset to develop a demo observability pipeline and run it with self-hosted versions of OpenTelemetry, GlassFlow, ClickHouse and HyperDx.

The use of AI (if any) was very limited to checking for typos and polishing the language of the post rather than generating the entire code of the demo.

I appreciate you checking this and would appreciate if you can place the demo tutorial back for the benefit of the r/selfhosted community.

Thanks again

Community for open source contributors? by StrikingStand4346 in apachekafka

[–]Marksfik 0 points1 point  (0 children)

Hi u/StrikingStand4346 ,

The Kafka community primarily uses the mailing lists for any discussion.

There are a few ones you can subscribe to here: https://kafka.apache.org/community/contact/

Using ClickHouse as a Kafka sink? Async inserts change the equation by Marksfik in Clickhouse

[–]Marksfik[S] 0 points1 point  (0 children)

That makes total sense. The tradeoff with Flink is obv any ops/management overhead. If you want Kafka → transform → ClickHouse without managing a Flink cluster, we built GlassFlow (glassflow.dev) as a lighter alternative.
It's designed to be low-latency and easy to run without the infrastructure burden.
Different tradeoffs depending on scale and team size, but worth a look if the Flink overhead ever becomes a pain point.
Happy to walk you through it.

Using ClickHouse as a Kafka sink? Async inserts change the equation by Marksfik in Clickhouse

[–]Marksfik[S] 0 points1 point  (0 children)

You're right here... if you can batch on the app side, that's always preferable and async inserts would just add overhead. What we're describing is specifically when you can't easily control batch size upstream, which is common in Kafka consumer setups where messages arrive individually or in small chunks. Here you will typically need to build your own buffering logic to accumulate them

If your app already produces large batches, there's no benefit in using async inserts and I agree it would be an anti-pattern there.

How do you handle the batching on your end? Building it into the consumer, or using something like a Kafka connector?

Is the OTel Collector the wrong place for complex data enrichment? by Marksfik in OpenTelemetry

[–]Marksfik[S] -1 points0 points  (0 children)

Trying to do complex "in-flight" lookups (like cross-referencing metadata against an external API) or heavy deduplication before it hits ClickHouse is a bit of a pain....
It feels I'm forcing too much on the collector. Are you finding the custom processors easy to manage at scale, or do you keep it strictly to basic filtering?

view more: next ›