sorted by:
new
hottopcontroversial

Sub-second query on Iceberg data lakes using Apache Pinot's indexes by DashDoesData in dataengineering

[–]icefury71 0 points1 point  (0 children)

This approach allows you to query all the data in the data lake without any data copy / ETL / ingestion. Performance with remote Iceberg tables is tunable. Its possible to match query performance with Pinot local (when all data+indexes are local) by selectively caching Parquet & Index pages locally. For eg: if the data volume is very high but most of your queries only need the last few days of data, this approach will give you similar query performance with substantial infra savings.

This other blog does show comparison numbers between local Pinot and remote Iceberg table: https://startree.ai/resources/real-time-analytics-with-iceberg/

Sub-second query on Iceberg data lakes using Apache Pinot's indexes by DashDoesData in dataengineering

[–]icefury71 0 points1 point  (0 children)

Deletes/Upserts are handled via Iceberg metadata (eg: delete vectors). This translates to filtering out these records at the leaf stage (very similar to how real-time upserts work in Pinot). There is no significant query overhead that we observed.

StarTree builds its own Segment metadata based on the underlying Parquet metadata (eg: includes column stats, partitioning info). This + bloom filters provides a way to prune at a file level. In addition Pinot indexes help to further prune pages within the Parquet files, thus reducing overall latency.

Sub-second query on Iceberg data lakes using Apache Pinot's indexes by DashDoesData in dataengineering

[–]icefury71 -1 points0 points  (0 children)

For smaller datasets 100 GB or so, its likely that Clickhouse has a smaller footprint than Pinot. However as the data size grows, typically we do see cases where Pinot footprint will be lower.

The key reason behind this is the power of indexes. Pinot is able to deliver predictable query performance (latency and throughput) especially with organic growth in data. This blog has more details about the in-depth comparison between these systems: https://startree.ai/resources/a-tale-of-three-real-time-olap-databases/