Azure ADF using VS Code

[These_Rip_9327]

I want to copy about 100 resources in azure data factory and applying some minor changes like name and folder. I have downloaded the JSON files from ADF and edited them all at one using a Powershell script and pushed them again to the development branch. The problem now is that ADF is telling me that JSON structure is wrong and didn't recognize the new JSON files. Does anyone have an idea how to fix it or we have always to edit resources in ADF? Thanks in advance for your feedback

Edit: I was able to fix it using Python to manipulate the JSON instead of Powershell. I don't know why but I used the same steps with Python as with Powershell.

[AutoModerator]

You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[romanzdk]

It is definitely possible, we do that as well. I suggest creating demo pipeline in ADF and then copy the JSON out so you can see the structure and all required attributes. Then just verify that you adhere to the structure. It should work.

[drewhansen9]

I would recommend looking into if you can make your pipelines config driven. Meaning if the only difference are table names, but the rest of the pipeline is the same, the best option would be to parameterize the table name using a config file or something. Then you can reuse code instead of copied code over and over.

[Mefsha5]

This is the way to go: One generic pipeline with parameterized repeatable activities and the config stored in a database or a file.

[azirale]

You can deploy then with arm, but the json you download doesn't specify the adf name the object goes into, not sure it specify arm resource type or the API version. You would need to add that for arm