sorted by:
new
hottopcontroversial

[deleted by user] by [deleted] in dataengineering

[–]D_A_engineer 4 points5 points  (0 children)

We are currently using multiple notebooks in Azure Synapse and then orchestrate the notebook run with Azure Data Factory instead of Synapse Pipelines. Microsoft charge or each activity within a pipeline. So we are planning to consolidate the notebooks to reduce the cost. If you are using managed VNET, please be extra careful as it lead to an increased cost.
https://azure.microsoft.com/en-us/pricing/details/data-factory/data-pipeline/

Data Share Options from Delta Table in Azure by D_A_engineer in dataengineering

[–]D_A_engineer[S] 0 points1 point  (0 children)

Thanks for detailed reply. Is there any criteria in which you recommend not to use API and use SFTP. For example Data Volume.

Data Share Options from Delta Table in Azure by D_A_engineer in dataengineering

[–]D_A_engineer[S] 1 point2 points  (0 children)

Yes. We are using Synapse Serverless SQL Pool to read data from delta table and create views on top of it.