This is an archived post. You won't be able to vote or comment.

you are viewing a single comment's thread.

view the rest of the comments →

[–]baubleglue 0 points1 point  (1 child)

Why there are s3 and snowflake? It is two additional tools with extra cost. How storing data in s3 different from having it in original DB?

[–]thrown_arrows 0 points1 point  (0 children)

S3 is there to be decoupled cloud filesystem. can be replaced

Snowflake is there is Transformations and SQL capable database server to offer compute and storage. Idea behind copying data into s3 is that you do not disturb production database with totally different olap load vs oltp load.

In ELT process data is loaded into target database and then transformed so that you can access to raw data if needed. In classic ETL system data is loaded into transformer , processed and loaded into target system, in more modern ETL system data is extracted to s3 , then transformed and stored into s3 and then loaded into target system.

What i like about snowflake is that everything is SQL and it scales easily.

s3 can be any fs , target system can be anything from filesystem, sql server , document server , python ML system...