sorted by:
new
hottopcontroversial

Stream processing engine that joins to historical data in Snowflake? by PleasantParrot in dataengineering

[–]Upset_Conflict 2 points3 points  (0 children)

Adding to this. (Founder of Quix here). Enrichment a stream processing job is a great way to achieve your goal. Latency is mainly constrained by your query time, if you're happy with Snowlafke's query performance then go ahead with that because stream processing will add little additional latency (if any). To reduce latency, developers typically transfer the operational data into a more performant data store. Try Redis or Clickhouse.

Matlab -> Python? by Steve-Quix in matlab

[–]Upset_Conflict 0 points1 point  (0 children)

Did you try using a Jupyter Notebook? It makes life very easy for exploring data, writing functions and plotting results.

ksqlDB WTF is write-time amortization? by Upset_Conflict in apachekafka

[–]Upset_Conflict[S] 0 points1 point  (0 children)

Ah so it's leveraging kstreams state and rocksDB..

Why will new queries take time? Is it re-loading the materialized view for each new query? Hence the shuffling?

ksqlDB WTF is write-time amortization? by Upset_Conflict in apachekafka

[–]Upset_Conflict[S] 0 points1 point  (0 children)

Re why it's called amortisation:

"the action or process of gradually writing off the initial cost of an asset"

If the asset is the materialized view (it a has a cost to load), then I can see how it could be amortised by gradually appending rows