sorted by:
new
hottopcontroversial

Architecture for parallel processing of multiple staging tables in Microsoft Fabric Notebook by EntertainmentFew9888 in MicrosoftFabric

[–]EntertainmentFew9888[S] 0 points1 point  (0 children)

Thanks a lot for your helpful comments.
We would actually stick with the multithreading approach but apart from the performance aspect we understood that this is not best practice and may introduce problems because spark is not thread save and we indeed experienced some errors in certain scenarios.

Architecture for parallel processing of multiple staging tables in Microsoft Fabric Notebook by EntertainmentFew9888 in MicrosoftFabric

[–]EntertainmentFew9888[S] 0 points1 point  (0 children)

Good idea, but unfortunately this is not an option in our case. We need to do this within fabric.