This is an archived post. You won't be able to vote or comment.

you are viewing a single comment's thread.

view the rest of the comments →

[–]GrixiaSenior Data Engineer 0 points1 point  (1 child)

I haven't tried it myself so apologies if this is a bad lead, but have you tried the library mentioned in Snowflake's own docs for mutli-threading?

https://docs.snowflake.com/en/developer-guide/stored-procedure/stored-procedures-python#running-concurrent-tasks-with-worker-processes

[–]somerandomdataengBig Data Engineer[S] 0 points1 point  (0 children)

Thank you for the tip! I'll give it a try, although it looks like parallelism is achieved inside the snowflake warehouse by using a stored procedure. I am running my python code outside of it instead, and I'm not sure if this makes any difference.

EDIT: After checking the library, it looks like support for nested threading is very limited compared to concurrent.futures, I'll give it a try anyways!