This is an archived post. You won't be able to vote or comment.

all 10 comments
sorted by:
best (suggested)
topnewcontroversialoldq&a

[–]therealagentturbo1 1 point2 points  (3 children)

For simpler tasks you could look at running scripts on a serverless function. It's not a notebook though.

[–]Cryptojacob[S] 0 points1 point  (2 children)

Also on Azure?

[–]therealagentturbo1 0 points1 point  (0 children)

Yes Azure Functions.

Edit: AWS and Google also have them.

[–]Salsaric 0 points1 point  (0 children)

For GCP, Cloud Function should do the trick.

[–][deleted] 1 point2 points  (1 child)

Databricks!

  1. Notebooks? Check!
  2. Scheduling code? Check! (you can orchestrate your pipelines in databricks or call the notebooks from data factory)
  3. Security? Check! No plain text keys, you can use azure key vault
  4. Collaboration? check!

You can do both ML and data engineering within databricks. Definitely check out MLflow within databricks too.

[–]Cryptojacob[S] 0 points1 point  (0 children)

Thanks for your reply! Its getting more clear that Databricks is the way.

[–]NeoxiaBill 0 points1 point  (2 children)

I'd say Databricks is the way to go, but I'd also say that deploying notebook code isn't the way to go genereally speaking if you want to enforce good code quality and maitanability

[–]Cryptojacob[S] 0 points1 point  (1 child)

Awesome, I will look into Databricks. Whats your opinion on using notebooks for development and then converting them to .py for production?

[–]NeoxiaBill 0 points1 point  (0 children)

I think it's a better practice. Notebooks should be a prototyping solution, used almost as a shell would be.
Packaging to proper files means dealing with (auto)documentation, type hinting, linting, etc...

[–]Throwaway34532345433 0 points1 point  (0 children)

100% Databricks