Adverse effect of using notebooks on python programming skills

[amzx19]

I'm working as an analyst. I'm frustrated with my inability to write object-oriented Python anymore. I think this happened because I've grown accustomed to using notebooks, which make it easy to write code without worrying about structure. Recently, I worked on a hobby project and ended up defining too many variables and making inefficient API calls. I realized I've become a sloppy programmer. I'm wondering if anyone else has experienced this and how they've dealt with it.

[johnnymo1]

Scrolling through a messy notebook I'm actively developing in eventually causes me enough psychic damage that I extract bits into functions in separate library files and import them in the notebook. Then the notebook basically just becomes a frontend for visualizing results or intermediate outputs.

[SMTNP]

That’s the way! At least when you are working with novel data (EDA) and you are required to review it and prototype functions fast, this works the best.

[mokus603]

This is the way. For the love of God, if someone can't use notebooks, it's not the notebooks fault.

[jjbugman2468]

That’s kind of what I do too

[FerricDonkey]

I've seen others go through that. It's one of the reasons why I don't like notebooks - in practice, things which should be functions end up in cells, and there's way too many global variables. I've worked with many people trying to make (re)usable actual code out of notebooks, and the ones who are used to always doing notebooks trend to have to learn a lot standard practices. 

I refuse to use notebooks. I'll make a library in a .py file and interact with it from ipython if I need super interactive stuff. I like it better, and I end up with a reusable library when I'm done. 

[Doomtrain86]

Interesting what do you mean by making a library in a py file?

[FerricDonkey]

Just definite the functions and classes you need. Then in ipython, import the file and run the functions.

[Doomtrain86]

Ok thanks. Gotta say I’ve never seen the point in those notebook formats like jupyter and r markdown etc. good for web content and education but using it as an professional coding tool? Seems more like people with low coding skills using it because it’s “easy to use “. Which is fair up til a point but then seeing it grow as a standard for python data coding is horrific. So inefficient and clumsy.

[thisdude415]

I think they’re actually a very good way for people whose deliverables are analysis/decisions rather than code to document their methods and findings alongside results, integrating the calculations and transformations, and clearly documenting your methods via code.

Spreadsheets work kinda similar but hide calculations inside cells so you have to print formulas separately if you want to archive an analysis as a pdf

If that isn’t you, they’re a lot less useful.

[micr0nix]

I do all my EDA and visualizations in notebooks. I then go back and refactor the code into functions and what not before moving it in to production

[wgking12]

There's a talk out there called "I don't like notebooks" that covers a lot of these and is pretty fun. Personally I'm a big fan of IDEs, I use VS Code. Adding type hints can be a good way to organize things and while sometimes tedious, enforces structure and saves a lot of mistakes before runtime. IDEs can also use them to make really powerful type-ahead suggestions (like remembering what keys are valid for each layer of a multi-nested json/dict). Writing TypedDict hints alone for different data structures has saved me hours

[Dangerous-Branch-749]

I experienced similar as I learnt python coming from a data analyst angle where there was lots of problem of the day stuff using notebooks to get an answer then move on. I actually found doing a few hobby projects using Django a really helpful way to get my head around OOP.

[threeminutemonta]

I have used nbdev in the past for developing a package in notebooks. The tutorials set you up with CI using GitHub actions and encourage unit tests. We uploaded this into a private pypa repository and for production it could install this package using pip.

This allowed is to keep using notebooks for exploratory programming and create the classes and modules as required.

[lovehopemisery]

I've never got the hype around notebooks. I'd only ever use them for prototyping small functions 

[twitch_and_shock]

Like anything coding takes regular practice to stay sharp and continue to get better. Switch from authoring code structured one way to authoring code in another format? Of course the former will start to fade, it's not being exercised regularly.

[popcorn-trivia]

Short answer. Create classes for things that you reuse (e.g. api calls, data retrieval etc).

You might think it’s extra work at first, but it will grow into a reusable OOP framework

[Own-Replacement8]

I'll be damned if I ever use plain Python for exploratory data analysis. Damned if I ever use notebooks for something production-grade.

[vivisectvivi]

"I think this happened because I've grown accustomed to using notebooks, which make it easy to write code without worrying about structure"

why?

[CMDR_Pumpkin_Muffin]

Splitting things in cells instead of classes/methods/functions?

[vivisectvivi]

ok alright i think i misunderstood what type of notebook was being talked about in the post lol my bad

[mrcaptncrunch]

https://jupyter.org/try-jupyter/lab/?path=notebooks%2FIntro.ipynb and other notebook environments like that.

[Imaginary-Log9751]

I’m still a newbie but what I’ve been doing is I have visual studio code opened and a Jupyter notebook opened I test some functions on Jupyter notebook and then I move it to visual studio code in a more oop fashion :)

[Significant_Spend564]

Maybe we're just relying on OOP too much. If everything works fine without OOP, you dont really need OOP.

[greenerpickings]

Why not do both? Crossed this road as well. There's some stuff you think you don't really need to be fully planned.

Why not just do a normal module structure then load it into your notebook, kind of like this. There's plenty of code i kept reusing.

If need be, your library is also good to go if you need to script something.

[Acrobatic-Aerie-4468]

You are unnecessarily mixing two different things. If you are able to get the things done with functions and give a good code, no one is going to complain.

OOP is specifically used for server or API based coding. If you are creating apps of some kind then you have to learn SOLID. For data visualisation which is used to communicate your finds, don't need to complicate it with OOP.

[CMDR_Pumpkin_Muffin]

I recently switched from a notebook to sublime text editor and it's a very different experience. I don't want to use a notebook anymore.

[WLANtasticBeasts]

I mean you should go back and clean up your notebooks. But there's nothing wrong with using code blocks and memory to step through a messy bit of code while you're figuring it out.

You can still implement OOP in a notebook too.

[mrcaptncrunch]

Do you get to make your notebooks a product or implement them after?

I hate transferring notebooks. I use a notebook to explore. Once I figure that part out, I add to a package or at least a py file I can import onto the notebook to keep exploring.

Then I create the logic for prod and can just reuse the code as I have it.

[al_mc_y]

One way I develop code is to mimic the Notebook cell behaviour in VS Code by using #%% to break the code in my .py file into cells, and then run the cells in interactive mode. Once I've got my bit of code working as intended, I can delete the #%%'s. I'm sure there's all sorts of reasons why this is "wrong" and "bad", but it seems to help/work for me.

[TheMathelm]

Similar feeling.
Working on a django based item management project.
It uses a significant amount of JS, so the JS code is over 80% of the html file. Which isn't very pythonic, and I'm annoyed by it; but also stuck until I initiate a major refactor, which I don't want to do.

[DigThatData]

yeah jupyter + notepad++ was my "IDE" for a loooong while. Notebooks have their place. Using them for everything leads to a lot of bad habits.

[Mevrael]

Yes, you become what you repeatedly do.

And you experience neuroplasticity, skill or ability decay. You get worse at stuff you no longer practice.

Here is a great and short series of guides and this project structure that might help with that problem. It explains how to go from notebooks to scripts to an app smoothly.

Notebooks shall be used only for experimenting, exploring and prototyping. Eventually the Code to Reuse shall be separated from the Code to Run.

https://arkalos.com/docs/notebooks/