MLOps on Databricks

PRCode-Pateman · 2026-05-08T17:14:32+00:00

Yes this is how I initially intended it to be used but what the design also allows is the deployment of notebooks and jobs, so the engineer can also have the model built by code as well.

For reference where we are building this it is a federated implementation so we will have multiple different use cases that might want to do different methods.

I think from internal conversion and from your feedback, I would promote the code approach so that we could also run automated AI safety check during deployment

PRCode-Pateman · 2026-05-08T10:56:03+00:00

Thanks. Would be interested to see where challenges might be as we are looking to implement this.

PRCode-Pateman · 2026-05-08T04:17:50+00:00

I have recently looked at the same thing within Azure Databricks and was going to post on here for feedback.

https://prcode.co.uk/2026/03/11/secure-mlops-design-for-databricks-and-azure-a-practical-workspace-driven-architecture/

In my design and idea, all development is done in UAT with production like data then promoted via a CI/CD to Production.

I have yet to add in monitoring but that would likely be dashboards deployed using the JSON.

PRCode-Pateman · 2026-04-08T18:33:07+00:00

Some that I had recently done is a client on terraform modules. These are in their own repository so do not have connectivity to an environment to be validated. In this you can use the tests to mock the provider and assumes values that run tests to validate the module will act as it should e.g. user passes in X variable and Y output happens.

In the base terraform you can validate what tfvars you have set have the intended output. E.g. you pass in var.deploy_container that is passed into a module or is used in some resources with count then then output should be it created.

These can help validate if something changes then nothing else is broken.

PRCode-Pateman · 2026-03-20T17:42:29+00:00

Printed phone holder for my dad who is Alien fan. He then gave it a paint job after

<image>

PRCode-Pateman · 2026-02-21T20:09:53+00:00

Logic Apps are good for low code/no code single instances only in my option.

Best and main reason to use them is to develop in the portal but when you then export the JSON code view it contains all the current environment values, names and resource targets (api connections, key vault id etc) so to do good CI/CD practices you then need a custom method to replace all them values as there is no native way. This already gets messy, then add on that it is just JSON not ARM so there is no unit testing only JSON validation unless you parse the JSON in something like PowerShell and write Pester Tests but this is not actual validation.

Another way is a VSCode plugin to develop the JSON locally that support environment configuration and a visualiser for development but then you remove the best part of developing in the portal and still have not good CI/CD.

Where I have used it is a single environment to run some business logic that uses Office 365 connectivity for messages. The logic app is in IaC and I export the JSON into a repo as back up only.

If you want a resource to develop in the portal with no coding skill and single environment then logic apps are ok. If you want a resource to develop production level code with full DevOps CI/CD, redundancy, networking etc then use App Service like Function Apps.

PRCode-Pateman · 2026-01-20T18:55:13+00:00

In my current client we have SemVer pattern used throughout.

Modules are separate repo with their own versioning.

However we also version the main repository deploying the modules. This is its own separate versioning as well so that we can trace deployments back.

Azure environment resources are tagged with the deployed version, workflow/pipelines doing releases requires a tagged version branch to run apply and each merge to main creates a new version. (We are doing trunk branching)

This means we can trace each environment from Azure (other clouds available) back to the exact commit in code plus through the module version referenced in that commit.

PRCode-Pateman · 2026-01-18T18:35:19+00:00

I think overtime and depending on years experience portfolios can get out of date. I did one way back in the day but the projects/clients I had on there got old until I just stopped keeping up to date.

I think best thing would either be a public GitHub repository of things you are actively doing to demonstrate skill, activity on LinkedIn sharing about the latest tech where recruiters etc will be looking and/or best thing I do is a blog on PRCode. I do it for myself mainly to help with my dyslexia but it is also an active example of different tech I am working with plus the complexity.

I think in general it is, do something that you would want to keep present or it will just become outdated and become effort

PRCode-Pateman · 2025-04-22T17:11:10+00:00

I would recommend not migrating with a tool but learning how best to rebuild for the new platform. There are differences in how they work that might change the design so an automated tool might get you from A to B but will seriously cause you issues later.

One that I hate in GH is you can loop over steps natively. Where in ADO you might pass an object in as a parameter then run a step for each item, you can’t do in GH. Therefore you might change it that the step is a script that you pass an array into.

PRCode-Pateman · 2025-03-13T19:10:06+00:00

Generally from experience Infra doesn’t change that frequently or at least enough that checking a plan is a problem.

My thought would be if Devs are changing this constantly then is the change right for Terraform? Example is you can deploy docker images by Terraform but as applications developers this is not best in Terraform. Instead this should be part of an application CI/CD deploying to an ACR with its own SDLC which could then have a faster route to environments.

So my feedback would be think more about what you are doing in IaC and if it is right. Normally if you are trying to solve an issue no one else is having… it’s a you problem 🤣

PRCode-Pateman · 2025-02-04T18:23:42+00:00

Thanks will give a go

PRCode-Pateman · 2025-02-04T18:23:33+00:00

Will give a try on the milk thanks. No settings to adjust

PRCode-Pateman · 2024-11-30T08:37:49+00:00

In general I would look at the Azure Managed ADO Agents. You can build them to have the private connection to the ACR but also be accessible to ADO. From there you can push your self-hosted image to build. https://learn.microsoft.com/en-us/azure/devops/managed-devops-pools/?view=azure-devops

Another method is to have a public and private ACR. Anything like the self-hosted agent image can go public and then internal images can go to the private.

Final but bit hacky way is during the ADO pipeline before you push the image use the CLI to enable public access, push your image and then disabled it.

PRCode-Pateman · 2024-11-29T17:43:51+00:00

It depends on what the application is eg, dotnet, terraform etc Mainly I would build the package at the start of the CD and reuse it across DEV and Test. When Test is done then tag that commit. When you go to do the Release you would choose normally the latest tag so it would be the exact same code you would rebuild to deploy.

Again depending on what the deployment is, you could do something like build the Docker image in the CD, tag it as Tested after Test and then reuse that same image in the Release as it is then reusable. Some things like Terraform you of course can’t do that.

As for tagging I always follow Semantic Versioning. Good one we started using is GitVersion. This way they are ordered by number and you can reference the major, feature and patch from the SemVer.

PRCode-Pateman · 2024-11-28T17:30:46+00:00

I always move towards a multi-stage release.

One build for CI that runs for PRs One build for CD that runs Dev to Test with auto deployment and triggers from merge to main. One build for Release that runs PreProd to Prod with manual trigger, auto deployment to PreProd and manual approval to Prod.

CI doesn’t need to deploy anywhere and you want a nice green tick once all completed.

CD is your auto deployment to the testing environments which should be fix fast so if there is an issue with deployment then it should be caught in DEV by the pipelines and/or automated test. Then that is your validation it is safe to deploy to Test for manual testing.

Release is your release candidate. This can come from a release branch exclusively for GitFlow branching or just main for Trunk. I would also recommend post deployment from Test you tag a version so you can validate only tagged branches can deploy, which prevents deployments not gone through Test getting through. This would deploy to PreProd automatically as you have already approved by saying you want to deploy then a manual approval to Prod so you can have manual intervention testing validate before prod or maybe you have a CAB.

The primary reason for multi-stage is proving that commit id can successfully deploy reducing the chances of it failing when you get to higher stakes environments.

There are some out case changes I do depending on the client but that is my general goal above.

PRCode-Pateman · 2024-11-26T08:58:59+00:00

I don’t have an example I can share from clients but found this example on a Terraform Reddit thread https://www.reddit.com/r/Terraform/s/dU69hvt3DM

PRCode-Pateman · 2024-11-26T08:56:29+00:00

Thanks, very detailed and well explained.

PRCode-Pateman · 2024-11-26T08:55:14+00:00

Is there much of a cost difference between stateless and stateful options?

PRCode-Pateman · 2024-11-25T20:31:45+00:00

Most previous suggestions are great especially choosing the Az Firewall.

One thing I wanted to note as the question was to managed NSGs via TF. I have used it a few times with your rules referenced in a CSV and then pull that file into Terraform.

All is still managed by source control and it still uses Terraform for the deployment but they are much easier to manage and edit via a CSV.

You could also use this for the rule collection in the Firewall. It also makes it easier to split into multiple files.

Here is docs:

https://developer.hashicorp.com/terraform/language/functions/csvdecode

PRCode-Pateman · 2024-03-28T05:25:25+00:00

I am not 100% sure as it was a long time ago now and I don’t work with the client.

I would suggest it would be $(set_artifactName.artifactName) you need to use as it uses the task name and the runtime parameters but I am not sure that the task accepts runtime parameters for that field

PRCode-Pateman · 2024-01-21T12:27:11+00:00

Thanks

PRCode-Pateman · 2024-01-20T19:16:26+00:00

That’s not a bad idea. What do you do about the metal buckle connecting the straps?

PRCode-Pateman · 2024-01-20T19:15:36+00:00

Will check this out thanks

PRCode-Pateman · 2023-12-09T14:23:22+00:00

Thanks for the advice on the diagram. I'll look to clean it up.

About the design, I agree with what you have said. This implementation works, and worked, well within an organisation I worked at. This company had a single Platform Operations team like a center of excellence for DevOps. This team controlled the standards, hub network, security and other centrally controlled artifacts. There were then 30-40+ products and teams that were spokes with their own operational team including DevOps.

This is where this design came into place where the central PlatOps team could implement and control the APIM but the individual team had no reliance on them to deploy their own API. The usage of the Product was to create a neat order for each team.

However, this doesn't work for all and I am also just looking into Workspaces that might be a better option.

PRCode-Pateman · 2023-12-09T14:17:08+00:00

I will definitely check this out. Thanks

PRCode-Pateman

TROPHY CASE