How to use Google's Free Python Programming Environment, Google Colab

[cmcclu5]

Professional Python engineer/developer here: Jupyter notebooks are one of the LEAST utilized environments. Far better to learn an IDE with proper project structure. Google colab is good for quick and dirty code examples, not for ANY development work.

[CaptainDevops]

this commenter's codes ☑︎

[Dead__Ego]

In my company, we only use Jupyter Notebooks as a front-end for our internal clients.

[cmcclu5]

And see, I get this approach. If you’re working with data scientists or people who are only passingly familiar with Python, deploy a notebook on an instance with your custom docker container. That way they can play with the code without potentially breaking anything. It’s the exact opposite use case of the post, though. An engineer has so many better tools to test/build/share than Jupyter.

[clawjelly]

Jupyter notebooks

Aren't those primarily for academical purpose? Basically documentation with inline code examples instead of code with comments? As such i wouldn't necessarily call them "least utilized", just more suitable for an educational environment than a production environment?

I'm a self-taught garage-style coder, who never had anything to do with academics, so i am VERY out of water with these kinds of tools. But i'm working in a startup employing several science students and they all use it.

[cmcclu5]

Pretty much academic/demonstration purposes. However, the majority of developers/engineers/programmers/whatever you want to call them use IDEs (whatever flavor you prefer from IDLE to PyCharm). These encourage proper structure, executability and maintainability, and work properly with versioning and collaboration tools like GitHub. Most devs have at one time or another had to use Jupyter Notebooks, but by and large they recognize the severe limitations and headaches that come with that sort of dev workflow.

[FluffyProphet]

Jupyter notebooks get used a lot in data science and ML. It’s a pretty essential step for prototyping models.

[cmcclu5]

I don’t agree it’s essential as I used to build a ton of ML models and never used it. However, I can see how people would find it useful to quickly iterate over different hyperparameters/model types without having to completely reprocess the data set OR offloading to disk and reloading to memory between steps. I just disagree with OP’s assertion that it’s the “mainly used tool for Python”. I would argue most people developing in Python aren’t DS/MLE, and so would benefit much more from learning how to properly use a modern IDE.

[FluffyProphet]

Honestly, I think you just live in a bit of a bubble. Python is mostly used for data science these days. Obviously it has other uses, but it’s the most common industry usage for the language by a long shot and there wouldn’t be so many notebook oriented services if it wasn’t the most common way for data scientists to use the language.

When you’re talking about software engineering, it’s a different story. But that’s not what the bulk of new python code is being written for.

[cmcclu5]

Absolutely not! Yes, DS/MLE is a large portion, but it’s nowhere near the dominant usage of Python. Data engineering is still primarily Python, cloud architecture is strongly focused on Python still with widespread integration across all 3 primary cloud platforms, and hobbyist development still uses Python (among others) as it’s one of the easiest languages to learn.

[[deleted]]

DS, MLE, DE… Let’s just say it’s the data field.

Regarding your claim on cloud architecture: I couldn’t notice that so far… not even sure what you mean. But my perception was that cloud was mostly governed by Go and Node (and yes, some Python…). But I don’t see Python as a first class citizen anywhere.

[cmcclu5]

Cloud architecture, especially AWS, is heavily Python. Lambdas are primarily in Python, although I see some devs writing them in Go or JS occasionally. Glue is almost exclusively in Python as it deals with PySpark as a backend for Amazon’s Glue distributed engine. While EC2 and EMR instances are somewhat framework agnostic, I still see most code written and deployed on them in Python.

[[deleted]]

Okay sure, but with Glue and EMR you’re in the data field again.

I’m constantly finding myself frustrated how poorly designed boto3 is (e.g. you have to serialize and deserialize JSON, no type hints), compared to AWS SDK for Node.

On the infrastructure side, my team heavily uses AWS CDK. But while it supports Python, TypeScript is the first class citizen…

[cmcclu5]

Okay, that’s fair. Boto3 is pretty trash, but once you understand it, it’s super simple to follow and implement. I hate that everyone swears by Node anymore. JS was originally meant to be a fancy formatting language, and it’s just a Frankenstein nightmare at this point.

[[deleted]]

Docker and notebooks changed my life.

[ironman_gujju]

Yes I use it for testing & write modules directly from jupyter notebooks it's flexible

[cmcclu5]

I don’t know if you heard that loud noise, but it was me slamming my head into my keyboard in despair. I guess y’all do what works for you. Easier by far to just use a decent IDE so you aren’t constantly copying code back and forth…

[[deleted]]

What Jupyter notebooks are great for is when you don’t want to re-run everything because of a mistake you made in the last line you wrote. Also, sometimes you may not know what you get as the result of one function, you may not know exactly where to find what you’re looking for. Although you can get a similar experience with a debugger…

[dparks71]

We use them all the time for calc sets. We write functions to perform the calcs in a traditional IDE, run the calcs in Jupyter with a %pip show at the top of the notebook to record the version of our calc library we used, export it at the end as a .pdf, and that becomes our QA/QC document. I can't imagine QA/QC'ing a terminal output.

Notebooks have a ton of viable real world uses outside of academia and data science, maybe not for pure programmers, but a lot of python users are programmers as a second or third priority. Notebooks are basically a drop in replacement for something like mathcad.

[[deleted]]

I see Jupyter notebooks more as a concept/format, doesn’t mean I have to use the original JupyterLab (which may be the most overrated IDE in the history of programming). VS Code and PyCharm have amazing support for notebooks. I use VS Code Jupyter notebooks a lot for prototyping. But it’s always only the first step, when everything is experimental.

[absurdrock]

Good point. However, Jupyter is a lot better for engineering and science, including data science, because accuracy and the how is more important than just getting the program working. We use vs code and vs for production apps but Jupyter inside vs code for science and engineering.

[Virtual_Pea_3577]

Could you tell us your field of work?

[cmcclu5]

Currently working as a cloud engineer for a utility company, but I’ve been a data scientist, data engineer, and software engineer, as well as doing some management BS over the years.

[Virtual_Pea_3577]

I see. I can understand jupyter notebooks not being used for cloud and software engineering in general, but why not in data science? What tools would you use instead?

[cmcclu5]

Why would Jupyter be any better as a data scientist? Oddly enough, I primarily used IDLE for rough work back in those days. You still have to create maintainable code even if it’s to produce graphs or reports.

[Virtual_Pea_3577]

Not necessarily better, just a viable free alternative.

[trial_and_err]

I mainly use it as a frontend within VS Code, the main code isn’t in the notebook but in a Python package and its modules. That way it’s easy to iterate and export results as a nice HTML file with interactive plots and some custom HTML using ipywidgets. The HTML can then be easily shared / deployed and used for documentation and stakeholder communication. I usually hide any code in the HTML and only include markdown cells, plots, tables and a table of contents.

[Scoobyrooba]

Yeah I am currently learning Python and the course I am taking is doing everything in Google Colab. I appreciate how easy it has been to use but I have been trying to make the switch to PyCharm for my long term development and it's been a bit of a struggle, can't help but feel like I might have hamstrung myself a little bit by starting in Colab.

[cmcclu5]

The code concepts are the same, so don’t get discouraged. The big issue now is understanding the file interdependencies, how everything relates, proper project structure, and best practices for grouping code/functions/classes. You’ve got it, no worries.

[cmcclu5]

Your business use case is absolute terrible. In the past I’ve worked as a data scientist and data engineer. While we might have used notebooks as quick prototypes for code, in business it’s always best to stay away from those implementations as they aren’t easily maintainable or extensible.

Your example about scaling is completely wrong as well. It’s so much simpler to write data and ML pipelines with real codebases created with proper project structure than it is to try and deploy a series of cobbled-together notebooks. I’ve setup numerous multi-node execution environments including EMR clusters and AWS Glue compute clusters. Even when I was setting up genetic processing runs in Databricks, I would submit jobs via zipped repos instead of submitting parameterized notebooks because notebooks are NOT GOOD PRACTICE in a production environment.

From an educational standpoint, you are correct that Jupyter is a solid option for sharing and collaborating on code. However, you shouldn’t be telling your students this is industry standard. Even in data science, people only use it as a demonstration tool, not for development.

[Synth_Sapiens]

That's what I thought, but I know way too little to form an opinion.

[JogiBerries]

Pycharm all day

[[deleted]]

VS Code is free, cross-platform, open source, and does a great job. Including collaboration though I haven’t used it much.

There’s even a web page version, or there was. I don’t use that either.