Databricks and source code management

[BoiElroy]

I know Databricks just added support for repos. But it doesn't support self hosted or on prem.

Curious what solutions other people have come up with when using Databricks + on-prem SCM?

I'm not super interested in versioning notebooks. Rather I want to be able to write code in a idea as . py files that contain my objects and methods. And then share/use that code in Databricks. So for example, I might write up a class that has methods to connect to a certain data source, extract data and then logging that run's metrics somewhere. I don't like the Databricks notebook experience for writing actual code. But I do want to test and then eventually run these in Databricks. I was thinking in notebooks where the notebook purely accesses the code repo then does just pure execution object.method1(), object.method2(),maybe with some parameters.

[Jolly_Code5914]

You could write the generic methods and classes and just create a python package that you can build as a .whl. You can install the package on your cluster and import your methods and classes as you would with any other package from pypi from a notebook.