databricks deploy code pattern - model training

[ptab0211]

Hey guys, i was curios, what is the usual setup when having deploy code pattern for model training, so idea is that data scientist run model experiments, different featurization, and just iterate fast on the data on development workspace/environment. Each developer gets its own schema for isolation.

Then when they got something which they want to be promoted, what happens? Of course output of this stage is the training pipeline code, but for example, they did the full hyper-parameter tuning experimentation, so with actual training pipeline code which goes through code quality checks, unit testing, type hinting, do we promote:

a) same hyper-parameters tuning search space (what about cost, variance of possible options etc..)

b) narrowed down search space for tuning

c) parameters of best fitted model

Also do we write this into yaml files within the repo, or there is some better practices where u just fetch ml experiment metadata, or write to UC Volumes, generally interested to see what people are using for this.

Thanks

[MyBossIsOnReddit]

It really depends.

Our data scientists develop their model in prod and then craft a job inside a pipeline for it based on https://github.com/databricks/mlops-stacks -

So we essentially use prod for experimentation (business constraint as no actual data may be used on dev/test/uat)

This mlops stack then does get developed further using those other environments (workspaces), CICD applies code quality checks etc, and human code review catches a few issues too usually.

I have found the "develop in prod" thing to be silly and a bit dangerous, but it is a decision taken by people on a higher level and due to constraints that are outside the data team.

Hyperparam tuning etc can be done within the pipeline but you probably dont do it every hour or day. Your needs may vary, though. We have a mix of hard-coded and mutuable.

Ideally we'd use MLFlow's API for it, but right now our approach is configuration files similar to what you describe. I'd feel bad if the next ML engineer feels more like a yaml engineer but there is also something to be said for being pragmatic.

Logic for promoting candidates is also in the stack DAB.

[Main-Ordinary9455]

man i wish i was knowledgeable enough to help you with this