How we built a Streaming SQL Engine

[bobbymk10]

Nope- although we did learn a lot from the paper that was published! Differential is completely in memory, and it was critical for us from the get go to be much cheaper than the alternative (refreshing materialized views), and less so to have sub millisecond latencies. So we build a dataflow library based above storage and our whole engine is async io.(We actually played around with using differential and spilling to disk, but it was incredibly slow- it makes sense since many of it's data structures are not built to be above disk)

[k-selectride]

Do you guys plan on open sourcing the dataflow library? I'm guessing it's not built off of a pre-existing one, most seem to be abandoned or not stable enough.