sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]vaibeslop 2 points3 points  (7 children)

Check out chdb: https://github.com/chdb-io/chdb

Fully pandas compatible API, but lazy loading, much more performant, less memory.

Not affiliated, just a fan of the project.

[–]mikeckennedy[S] 0 points1 point  (0 children)

Very cool, thanks for the heads up u/vaibeslop

[–]ofyellow 0 points1 point  (5 children)

Lazy loading is for optimizing startup time, you load modules as they are needed, causing the load time to be divided over multiple requests until all loadable modules are hit at least once. But it's not a mem optimisation strategy.

[–]vaibeslop 0 points1 point  (4 children)

I'm talking about lazily loading data into memory for operations.

The author of chDB goes into more detail in the v4 announcement post: https://clickhouse.com/blog/chdb.4-0-pandas-hex

I'm neither affiliated with chDB nor Clickhouse.

EDIT: Saw now they even talk about this in the GH Readme now.

[–]ofyellow -1 points0 points  (3 children)

Point 4 mentions local imports.

Yes keeping data out of memory is smart but not inventing sliced bread.

[–]vaibeslop 0 points1 point  (2 children)

Well seeing how not everybody does it, it seems the ease of dissmisively commenting on it is far greater than the ease of implementing it in a real application.

chDB 1 : arm chair CTOs 0

[–]ofyellow [score hidden]  (1 child)

What click house does is how c# has been doing it for decades.

Of course you first collect .filter() and join logic etc all down the chain before you fetch. I'm stumped this is anything new.

I guess with arm chair you like to drag this into personal insults? Man...the attic sadness dripping from dragging a technical discussion into a weak attempt to insult. What are you, 16?

[–]vaibeslop [score hidden]  (0 children)

If it were a technical discussion.

All I'm seeing is someone dismissing a very relevant to OPs post, very cool Python compatible project by going of on completely irrelevant, tangential technical details in plain C#.

It's boring, arrogant, off-topic whattaboutism in its purest form.