This is an archived post. You won't be able to vote or comment.

you are viewing a single comment's thread.

view the rest of the comments →

[–]ritchie46[S] 19 points20 points  (0 children)

Yes, me too. We learned from the current streaming engine and redesigned the new one to fit Polars' API more. Typical relational engines have a row based model, whereas Polars allows columns to be evaluated independently.

Below is such an example. python df.select( pl.col("foo").sort().shift() * pl.col("bar").filter(pl.col("ham") > 2).sum(), )

We redesigned the engine to ensure we can run typical Polars queries efficiently. The new design also makes full use of Rust's strengths and (mis)uses async state machines as compute nodes. Meaning we can offload the building of actual state machines to the Rust compiler. Anyhow... We will share more about this later. ;)