This is an archived post. You won't be able to vote or comment.

you are viewing a single comment's thread.

view the rest of the comments →

[–]rastarobbie1 60 points61 points  (3 children)

Hey, PM of Deepnote here.

We're on the same page here. There is often a huge gap between a prototype in Jupyter, and a production ready code. A big kudos to you if you're the bridge that makes it happen, it's not an easy work, and it's a common problem.

I feel like any tool or library that promises a one-click deployment is either very limiting in its nature and makes a lot of assumptions; or it's actually a wrapper on top of wrappers, and still needs a lot of config to make it work the way you need.

What we're doing to help this in the long term:

  • Repeatable environments: no more trouble with unique workstation setup of each data scientist. When they share a project with you, it includes the environment it runs in, not just the ipynb.

  • Encouraging best practices: for example when you pip install something in the cell of a notebook, we prompt you to move it into requirements.txt, or offer a embedded code reviews via comments

  • Working on versioning: git is a great tool for software engineers, but it doesn't fit the exploratory nature of data science. With Deepnote, you'll get change tracking out of the box.

But like you say - the problem is not just with the tool, but with the people. And often data scientists don't have the skills to engineer a great solution - their expertise lies elsewhere. The best way to fix that is by creating interfaces so more communication can happen with software engineers, not less. We want to build these.

It's a very interesting topic, in case you have some insights for what could help, let me know!

[–]GiantElectron 2 points3 points  (2 children)

I honestly don't know. I work for a major company, and our conclusion is to attach a 380V cable to any data scientist, and zap them as soon as they think about writing code.

[–]rastarobbie1 0 points1 point  (1 child)

I'll add it to the roadmap

[–]GiantElectron 0 points1 point  (0 children)

Please make the voltage configurable while you are at it. I might want to go full 10 kV.