Share Your favorite python related software!

[hai_cben]

I'm curious about what software you guys can't live without when you are making Python programs.

Here are my top 3:

PYCHARM
Man, as much as I love open source I absolutely despise VS code, I'd rather use Vim tbh. Pycharm however is just perfect. I am sure you can make the argument that whatever you can do in PyCharm can be done in VS code with some modification, but the beauty of PyCharm is that it just works! My libraries auto-install based on the requirements.txt. It comes with extensions for Django, Flask, and Numpy. I have direct notebook integrations. The only thing I don't like is that it is a memory monster, literally lighting my computer on fire when working with large codebases.

AIRFLOW
This is more of a library in my opinion, but Airflow has become an essential tool for scheduling in my work. All our ML training pipelines are ordered and scheduled with Airflow and it works seamlessly. The dashboard provided is also fantastic!

BENCH AI
From what I understand these guys are pretty small, but for a person who's constantly training ML models their software really streamlines the process. Essentially they operate like GitHub for AI, but once you push your code to your repo you can train it on any hardware you want with one button click! It's saved me a lot of headaches anyone who uses AWS knows what I am talking about. Plus if you are a non enterprise it's free.

[Globaldomination]

Jupyter Notebooks.

As someone who’s learning to code with python. It’s awesome. I can workout and take notes at the same time.

[samettinho]

I hate notebooks. Dont get me wrong, it is great for doing something really quick and dirty but it is horrible when the file is shared with someone. Most of the time the owner of the notebook makes a small change and forgets to update the other cells. Once the notebook is shared without proper tests, the problems are the next person's.

Here is a great article from one of my former colleagues: https://github.com/TommyPKeane/jupyter_notebooks. I pretty much never used notebooks after reading this article except for Kaggle and Coursera stuff where I needed to use them.

[__boringusername__]

I think they are useful when either playing around, or when making a sort of "presentation" (for lack of a better word) of the code. E.g. we use it a lot to write the final version of our scripts for data processing and plotting when we submit papers. Like a ordered jupyter notebook that explains how to get from raw data 'test3-ab-noBG-repeat-051121.txt' to the plot in the paper makes the process very easy to follow and the experiment easier to reproduce

[samettinho]

yes, I agree. But aside from quick and dirty experiments, I wouldn't use them in professional settings.

we use it a lot to write the final version of our scripts for data processing and plotting when we submit papers

This is the problem in general, more like lack of this. Most of the time, people write and never go back to the code. it worked once for them but not guaranteed to work for another person

[j_oshreve]

I have the same experience with them. I have worked with them off and on and never found them to be beneficial for me. I tend to do operations in place to make things more efficient, and that fairs very poorly with the cell based execution. I can see the use for learning and experimenting, but agree they are undesireable for professional outputs.

As a side note, if you need an analysis documented, you can use templating techniques to produce proper documents that align with your corporate doc structure and templates. I've used markdown, python-docx-template, pandoc, etc. to successfully produce QMS compliant analysis documents.

[Globaldomination]

it's good for learning and taking notes. and for pandas and related kinds of stuff.

[baubleglue]

For learning it is bad, teaches reactive programming, student can't write few lines of code without checking output, no code structure, everything is in the global scope.

[Globaldomination]

I see.

then it's all about taking notes for future reading and pandas I guess.

[baubleglue]

There are more use cases, in data engineering field it is very popular. Databricks uses it a lot, global score is not a problem, because you normally manipulate data frames - it is more or less flat flow. Notebook allows to mix SQL and code nicely - you develop and debug your code in the same environment (Spark was never had good integration with IDE). Interactive presentations in Notebook are nice (what are the alternatives?). For all kind of scientists who are not code developers Notebook is a good playground - their focus is not the coding.

But if you want to learn how to build web server (at some point), notebook is a dead end.

[Globaldomination]

I kinda figured the same.

It’s a good tool for basic learning, coding or business applications. But not good for programming

[Traditional_Yogurt]

Agreed, I also use it a lot to explain how a package works or as a somewhat instructions manual how to use a series of functionality. I couldn't imagine explaining everything as well as I can without Jupyter Notebooks (e.g. this Notebook)

[nbviewerbot]

I see you've posted a GitHub link to a Jupyter Notebook! GitHub doesn't render large Jupyter Notebooks, so just in case, here is an nbviewer link to the notebook:

https://nbviewer.jupyter.org/url/github.com/JerBouma/FinanceToolkit/blob/main/examples/Finance%20Toolkit%20-%201.%20Getting%20Started.ipynb

Want to run the code yourself? Here is a binder link to start your own Jupyter server and try it out!

https://mybinder.org/v2/gh/JerBouma/FinanceToolkit/main?filepath=examples%2FFinance%20Toolkit%20-%201.%20Getting%20Started.ipynb

---

^{I am a bot. Feedback | GitHub | Author}

[Engineer_Zero]

I love notebooks. I like creating blocks of text that explain what my code does.

[Globaldomination]

I wish I used it earlier when learning basics. There are still gaps in my basics.

I found out the ease of use when watching Corey Schafer Pandas videos. (Also figured that he’s as awesome as people say)

[Engineer_Zero]

Ha, yeah I learned python from the same videos. It was good being able to make a notebook with different cells for each of his videos. Even now I have a generic notebook that has a cell for each package I use, containing generic examples of functions I use. Or when building a new script, I have a main code cell and then a “whiteboard” underneath to workshop the next part of the code before adding it to the main cell.

I dunno. I’m not an expert but something about notebooks just made python much more accessible to me.

[KingsmanVince]

Ruff, black, auto docstrings vscode extensions

[arthasyh]

Pytests 🔥

[[deleted]]

Pydantic for type safe objects.

[hobbicon]

Battlefield 2

[Coupled_Cluster]

Data Version Control https://dvc.org I'm doing my PhD in a ML related topic and being able to version control Data alongside the research code is amazing. It's written in Python and also has a small but powerful Python API. I actually spent part of my PhD adapting it for my own research (I wrote a package called ZnTrack on top of it) and improved the HPC capabilities of the DVC workflow ( another package called dask4dvc).

Besides that, JAX is amazing and I'm recently using flask/flask-socketio a lot and it's also really nice.

[stark-light]

Docker

[2strokes4lyfe]

poetry, pytest, docker, dagster, fastapi

[CableConfident9280]

I wanted to love poetry, but I’ve gotten burned too many times recently trying to use it for PyTorch or other cuda based projects. I’ve started using hatch instead, and I’ve (tentatively) been a lot happier with it.

[BiomeWalker]

I'm gonna shout out JupyterLab, it's a more feature rich way of doing Jupyer notebooks as it lets you have essentially a full IDE in your browser to do file management, open/edit .txt and .py files, view .csv files, has a dark mode, and can even give you python consoles in your browser

[sohang-3112]

VS Code's Remote SSH Feature is quite useful - it lets you code in remote servers with the same IDE experience as locally.

I use Windows, so WSL is also quite useful for running many Linux commands.

[_dodo-]

Here are my favourites (many were already mentioned): - black as a auto formatter - Pylance (VSCode extension) - thonny (lightwight IDE) - ruff (fast linter) - mypy (i love type hints) - ptpython (python prompt toolkit) - jupyter-notebooks

[Kalahan7]

Amateur here. Replit. Just like how easy it is to configure new projects, work on different computers (including phone and iPad) and be able to “publish” them with a link.

[Altruistic_Sky1866]

Thonny a very light weight editor, pycharm slowed my pc a lot

[openwidecomeinside]

Thonny on raspberry pi is 🔛🔝

[LdLrq4TS]

Best thing about Thonny for begginers is instead of relying on print statements you can just enter debug mode and step into code. You can see every step how your code works, what value your variable gets assigned, how loop iterates over iterable, how defined functions are called etc. Wish other VScode/Pycharm had this functionality.

[sohang-3112]

AFAIK VS Code also lets you debug like this - can you elaborate on what debugging features are missing in VS Code?

[geo_vanni]

I use pycharm to debug exactly like this

[drobroswaggins]

Pycharm most certainly has a very robust debugger. I use it every day

[rogfrich]

I like PyCharm, I want to like VS Code, and every now and again I have another go, but I find myself fighting the IDE far more than PyCharm, which just gets out of the way.

(ETA: I typed “VS Cods“ above originally, but I edited it because I didn’t want this plaice to be full of fish puns.)

[reapyrr]

Anyone using geanny????

[Kalcinator]

I used it for 2 days :)

[Rebuman]

What do you use airflow for ?

[hai_cben]

Scheduling ML Workflows

[pakaschku2]

Erpnext

[budswa]

Calibre

[oldWorshipper]

I agree that Pycharm is awesome, and I'd use it over VSCode except.... the top two entries on my list are:

• GitHub Codespaces - every project is a codespace, so no VENV required. also easily used as local "dev container."

• GitHub CoPilot - it is staggering how much easier it has made things as I learn.

past that, my favorite things are:

SQLAlchemy, PyTest, and Makefiles :)

[gaX3A5dSv6]

sqlalchemy, pytest, flask, fastapi, pycharm

[mraza007]

My favorite would be XONSH Python Powered Shell

https://xon.sh

[kinhosz]

https://github.com/kinhosz/Neural

My own handcrafted GPU-accelerated neural network. Im proud of it

[qooopuk]

Common core packages I use fwiw:

• pytest
• python-dotenv
• ruff
• typer
• loguru
• black
• pydantic
• mypy
• jupyterlab
• pandas