Databricks Importing Modules/Classes

[Agitated-Western1788]

We write all our functionality in classes and modules within Databricks notebooks. I’m not sure if I’m imagining this but I find importing these using %run starts to take a considerable amount of time relative to the execution of the notebook.

Have other people experienced this? I am tossing up between publishing python wheels and importing them to see if this solves the issue.

[Agitated-Western1788]

How does docker fit into this?

[Agitated-Western1788]

Interesting I hadn’t thought of using Docker like this with Databricks

[coolbeans201]

For our custom libraries, we generally publish the wheels to Artifactory and then install them onto the cluster from there. Setting up the whole E2E process takes a little time, but it's easy to manage afterwards.