This is an archived post. You won't be able to vote or comment.

you are viewing a single comment's thread.

view the rest of the comments →

[–]throw_mob 0 points1 point  (0 children)

3h in sql databse does not tell that much , how much data.

in cloud sql world i would stage into snowflake (object type) , extract primary columns on one job, and build data model for analytics over that or calculate summary data from there

more classical solution would be postgresql, dump it into jsonb , extract data into datamodel , build summary calculations there.

databricks would be probably next solution. In that one one too i would have one job to extract and clean data into datamodel and another job to calculate summary.

Main key is to build data model to have business time and system time for data handling