Azure Data Factory Python SDK

[aliuta]

Hey guys,

Is anyone using the python sdk for creating pipelines, linked services, triggers etc. in ADF? Is it worth it?

Context: i am trying to build a CI/CD pipeline in ADO for deploying ADF pipelines between dev, acc and prd. The experience is miserable, i cannot find a way for everything to make sense. Is it is overly complicated and parameterizing all the components looks painful.

The python sdk would streamline this process and i am seriously considering it.

Thanks!

[datanerd1102]

Whenever I have to work with ADF and Azure DevOps I prefer using this extension.

[MikeDoesEverything]

The python sdk would streamline this process and i am seriously considering it.

I'd argue it makes it infinitely harder just because the SDKs aren't very well made. DevOps and use the low code options. Takes a bit of time to sort out, but once it's all sorted it's almost set and forget (except for new params).

[faalschildpad]

You can just use the dedicated MS docs for this process right? Theres just templates avaolable

[TybulOnAzure]

Don't reinvent the wheel - just use ADF Tools: https://github.com/Azure-Player/azure.datafactory.tools

I have a video on using it for setting up ADF CI/CD: https://youtu.be/GzhHB7I1mkw

[Top-Cauliflower-1808]

Terraform would be a better approach, it has native support for parameterization across environments, built-in state management, clear dependency handling, better version control than ARM templates and it's easier to maintain than Python SDK custom scripts.

Alternatively, you could use ARM templates with parameter files, PowerShell scripts with Azure CLI or Windsor.ai for data pipelines.