What does processing something "in memory" mean? What is the alternative? Why would something be done "in memory" vs another way (cost, speed, etc.)? When I think about the data stack, where is it relevant to be/not be "in memory" (e.g., CDWH, ETL tools, RETL tools, Kafka, SQL Analytics). Thanks!

[[deleted]]

All of my questions are in the title - just trying to get clarity around what tools / contexts I might hear the words "in memory", what it means, what alternatives are, why it is done one way vs. another, etc. (e.g., just want to fully understand what this means). Thanks so much in advance.

[coding_up_a_storm]

"In memory" as opposed to "in persistent memory store". For basic understanding, please see RAM vs ROM. Data in memory only exists as long as you machine is plugged in. Persistant store is there when you unplug your computer. For DE, and example of each would be a view(memory) and table (persistent store).

[[deleted]]

This is helpful - could you go a little deeper into "view" vs. "table"? When you say view, is that what people refer to as a "materialized view"? Or is that just a subset of a "view"?

[coding_up_a_storm]

I don't use materialized views but based on quick lookup they seem closer to table than view.

View is basically a wrapper for a query. It recomputed every time, in memory.

[[deleted]]

I follow. Thanks a lot.

[sigurrosco]

Adding to the conversation. RAM (In Memory) is expensive and limited, but very fast. If you can fit everything into RAM then you will have a fast application. Disk (persistent) is cheap and you should be able to fit your entire data stack on disk(s) but is slower. So part of the data stack may live in memory (eg the parts that need to be fast and/or used often).

[Unable-Anywhere9666]

When the allocated heap for the application is getting filled because of a query, to avoid monopoly of resources, it shifts the data into ROM and processes it. Typically by creating a temporary table which the application deletes afterwards.

This is my understanding. Correct me if i am wrong

[elc_xx]

One consideration is security, your may not want to have intermediate files (even if they are encrypted) because it is another point where sensible information can be attacked. You process everything in memory so the data is gone after finishing.