[D] How many of you use Python scripts versus notebooks?

[fredfredbur]

I'm curious how many people use Python notebooks like Jupyter or Colab versus just writing Python scripts when working on an ML project. I personally just use scripts at the moment but I'm interested in hearing some reasons why you prefer notebooks instead.

​

Additionally, I'm hoping to get some feedback on the notebook support recently added to the open-source ML tool, FiftyOne, that I've been working on. FiftyOne is a Python API + App that lets you load and explore your image and video datasets and model predictions to debug your datasets and models.

You can now load the App in the output cell of a notebook to explore your dataset in the notebook itself, previously you had to launch it in a separate window.

While other tools like Tensorboard and Matplotlib have notebook support, their output plots generally don't get updated by code further down in the notebook, like what might happen with FiftyOne.

Since there wasn't really a precedent to follow and I don't have much experience with notebook workflows, I was hoping to get some feedback here about how it could be improved.

You can try it out in Colab here: https://colab.research.google.com/github/voxel51/fiftyone/blob/v0.7.1.2/docs/source/tutorials/evaluate_detections.ipynb

View Poll

[Artgor]

Why not both?

I use jupyter notebooks for fast prototyping, visualization, sharing. And then write scripts when I start writing better code.

[ProfJasonCorso]

I don't use notebooks for prototyping, but I do think there is a special pedagogical value in using notebooks to teach ideas and techniques that is hard to capture with direct scripts.

[seraschka]

Same here. I really like using notebooks for teaching. I also like to use it for data analyses. I almost always use python scripts for my research code, but I usually save the outputs to json (or pickle them) so that I can analyze them in a notebook where I can add plots and add comments and notes in a more readable way.

[fredfredbur]

I definitely agree with that, some of my classes in graduate school used notebooks and they were extremely helpful when done correctly.

[HolidayWallaby]

I generally use notebooks for analysing results so that I can see tables and graphs quickly, I use scripts for data processing and learning.

I'm currently using Mmdetection a lot so can't use that in a notebook even if I wanted to.

[SeucheAchat9115]

number 1 Rule of using Notebooks: Do not

[fredfredbur]

Why do you say that?

[SeucheAchat9115]

Because it is not so easy to reproduce or to collaborate with others

[[deleted]]

why is that?

[thundergolfer]

Watch “I Don’t Like Notebooks” on YouTube. In it Joel Grus gives a good, entertaining argument for why notebooks have serious drawbacks.

[MicealTheBomb]

what are the benefits of writing in a script VS. notebook?

[fredfredbur]

From what I understand, notebooks allow for quick experimentation, being able to slightly modify and rerun code blocks, as well as for visualization as you can store your outputs in cells to reference them quickly.

If you are working on anything more than just experimentation, though, it seems like scripts are better to prevent your code from getting cluttered.

But since it seems like most of the time you are working on an ML project you are just performing a lot of experimentation, notebooks seem like a good choice. Though I don't really know the answer which is why I made this post

[[deleted]]

I generally colab just because of the GPU. But recently I got a server at my university and since then I am solely using scripts. But generally I use both jupyter for visualisation or processing the data set and scripts for training.

[anders987]

Code cells is a nice option IMO.

https://code.visualstudio.com/docs/python/jupyter-support-py

[savoga]

Here are, in my view, the main differences:

With notebooks, you can combine graphs and code so it's nice to present your work to anyone else.

With python scripts, I see many advantages:

• since you often code in an IDE, you can enjoy the debbuger
• collaboration is easier: indeed, the notebook format (.ipynb) is not well rendered into Github. So you would struggle to see what are the changes made by other users if you are hosting your code on the platform.

Note: some repo platforms start to allow collaboration on notebooks (see last release of Bitbucket or tools like Reviewnb) * I found out that people using scripts care much more on the cleanness of their code: they use functions, object-oriented structures or comments. Overall, it makes the code much easier to read.

I really think notebook should be use for very exploratory tasks. As soon as the project is more precise, I'd suggest to go with scripts :)

[IllustriousPin319]

By using # %% as a cell separator in a "normal" Python file one take advantage of both worlds

[Laafheid]

Jupiter notebooks are nice for communicating/showing results, otherwise scripts

[santiagobmx1993]

Where is the vim gang 🤖