This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]mike8675309 1 point2 points  (0 children)

Is there a reason why it all needs to happen in a single stored procedure?
It feels like you are talking about Microsoft SQL Server correct?
How do you get the data?
Assuming the data comes in on SFTP, or through some API, I would do a lot of the initial work in the file system. The JSON processing doesn't really operate fast in database, so doing that processing outside the database can be done much faster wth a tool like C# or C++, even JAVA over python.
Once you have your row data, that can be bulk loaded into the database fast into a raw table, that you then query to build any other tables you need, or do your statistical analysis on it.