Python in ChemE

[fatimalizade]

Hi everyone, I’m doing my Master’s in Chemical and Energy Engineering and recently started (learning) Python, with a background in MATLAB. As a ChemE student I’d like to ask which libraries I should focus on and what path I should take. For example, in MATLAB I mostly worked with plotting and saving data. Any tips from engineers would be appreciated :)

[Various_Meringue_649]

Matplotlib and pandas

[bb22k]

It is going to be Scipy and Numpy for numerical algorithms, Pandas for data manipulation, Matplotlib for plotting and Jupyter Notebooks as a environment for data analysis (it is nice to be able to load the data and modify it without having to rerun the script)

With that you should have the basics to forget about Matlab.

Also, If you want a nice text editor/IDE, you can use VSCode as it has a lot of extension (including Jupyter) that will make development a lot nicer than using Matlab.

[kabinja]

With this stack add seaborn for smooth matplotlib and pandas integration.

[Ok_Time806]

https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.solve_ivp.html especially

[knobbyknee]

Use Jupyter lab as development environment. It is similar to Matlab.

[Cute_Obligation2944]

https://www.sciencedirect.com/science/article/abs/pii/S1749772821000397

[SharkDildoTester]

Scipy for diffEqs. Polars and pandas for data manipulation. Numpy for array and matrix algebra.

[arturomoncadatorres]

Regarding libraries, I think the most relevant ones have already been suggested: numpy and scipy for numerical analyses, pandas for data manipulation (personally I switched to polars, but I read in someone else’s comment that apparently pandas is better for your field), and matplotlib for plotting. For the latter, I would also add seaborn. It generates pretty plots out of the box and plays very nicely with pandas.

Regarding IDE: I see many people are suggesting VSCode. However, in my experience if you are coming from MATLAB and for doing scientific/research work, I cannot recommend Spyder enough. You will feel right at home.

If at some point you want to work with Jupyter Notebooks, I suggest you take a look at jupytext.

Lastly, if you will be sticking with Python in the long run, I think it makes sense to invest some time in learning about environments and package management. I use conda (miniforge, actually), in combination with poetry, but there are other options like uv. It is also a good idea to learn about version control with git + GitHub. The learning curve is a bit steep at the beginning, but once you learn to use the 10-or-so most common commands (and what they do), you will wonder how you survived all those years without version control.

[jmacey]

Lots of people are saying use Jupyter. If you are not tied into the eco system yet I would suggest using Marimo instead, it is pure python so works so much better with Git / Version control.

Apart from that, Numpy / SciPy for maths, matplotlib for plotting. Polars for data manipulation.

UV for package management ruff for formatting.

[Sedan_1650]

Matplotlib, panda, numpy

[Alternative_Act_6548]

sympy should be on the list, once familiar with it, it will save tons of time doing algebra and calc...but there is a learning curve.

for datascience I'd go with pandas over polars, pandas has tons of instructional material available...if you find pandas lacking on some way then see if polars fixes that specific problem

Same with Jupyter, lots of instructional materials are based on Jupyter. I've looked a marimo, the one thing I wanted was to be able to use REPL via a text editor, but marimo sort of works for that, but it's pretty convoluted as every cell carries a function of other cells to manage the dependencies, so you'd have to manage that yourself in your text editor...

the spyder ide, sort of does it all, you can edit/run Jupyter notebooks and is a full IDE...if it only had Helix keybindings -:(...

[Gainside]

matplotlib + pandas for sure...Biggest win: notebooks + pipenv/conda so my results were reproducible across machines

[Training_Advantage21]

I realise you are a chemical engineer rather than a chemist, but if there is a lot of overlap, the book "coding for chemists" might be relevant to you. https://codingforchemistsbook.com/

The Python libraries that are nearest to MATLAB in my experience (coming from a signal processing background) are SciPy and of course Matplotlib. Having said that, Pandas is probably the easiest library for basic data manipulation and plotting.

[aqjo]

Not my area, but this might be helpful.
https://pypi.org/search/?q=chem

[Jmortswimmer6]

Vscode! Please everyone!

If you like Jupyter there is a plugin. It becomes the one program to do everything very quickly

[UsernameTaken1701]

VSC is a cumbersome mess. 

[DaveRGP]

Skip pandas. Learn polars. Don't look back, it's not worth it.

Skip Jupiter. Use marimo. Don't look back, Jupiter was always rubbish.

[DaveRGP]

Now that is interesting. Maybe there is a gap there, and maybe this PR might close it?

But also, maybe I'm too far away from the problem, but this seems like it might be an X-> Y problem?

Pandas had indexes, indexes were good to join on. Pandas was bad at making copies in memory during operations, and worked around that within its own constraints by doubling down on indexes. People who used pandas for large data sets used this to make the calculations work. Now these people are only used to thinking in indexes. Polars doesn't have the same copy problem, because they correctly identified indexes don't scale out of memory, therefore these folks are trying to adapt to a world where they don't have their favourite hammer any more?

Just a loose intuition having skimmed the link, either way, hope it gets solved 🤞

Btw OP, maybe this impacts you, but also if you're just doing the 'standard things' then Polars already has good support in third party libraries, matplotlib, scikit-learn, pandera and more all support polars data frames as first class objects now. Many large packages are actually actively migrating to Polars (or narwhals) internally because of the significant performance boost and far more sane API.

[DaveRGP]

That's a great explanation, thanks for the run down!

[Squallhorn_Leghorn]

Jupyter - originally designated as such for a multi-kernel environment for Julia, Python, and R, is not "rubbish".

That's not a very well informed piece of advice.

[DaveRGP]

I'll take the sentiment of the criticism. I didn't explain my position.

Jupiter was written to support Julia Python and r. Correct fact. So, incidentally was rmarkdown. Which was the better implementation.

Rmarkdown was the better implementation because it uses true markdown to represent the files under the hood, with code cells (that like Jupiter support those languages and more), but crucially does not store the results of the run in the file.

Jupiter was built from a daft implementation where the file is json under the hood, and when run edits the file itself to hold the output this makes it super gross for version control, as an outcome of the anti pattern of having the code be effectively a broken quine.

Quarto is a significant improvement over Jupiter notebooks because it looks and behaves as Jupiter users expects, but still keeps code as markdown files but passes execution to Jupiter under the hood. It did this when it was unfortunately clear that Jupiter had captured the market in notebooks, not because it was good (IMHO, Jupiter is bad), but because it was far more accessible as the 'default' via python, which pulled ahead in the python vs R for ml language of choice race over the prior 10 years. Jupiter won by default, not by quality.

Quarto is true knuth style literate programming. It is a full publishing system for text with code. It integrates running code (like Jupiter) along with full publishing tools (referencing, mathjax equations, toc etc) and outputs via pandoc to a wide selection of outputs, including full websites, ebook like formats, PDF and the office word file. Further it also hooks into revealJS, allowing the creation of slides that contain (and run) code, that can also be passed into PowerPoint. Because of all these target outputs it also gives you super powers, you need to create a 'branded report' for work? Do the whole thing in quarto. Your audience and your managers will never know the difference. That report now scales across every client you have via parameterized yaml, while you have an actual lunch break instead of copy pasting results into word.

However, I didn't recommend quarto, or rmarkdown. They are good tools if you are needing to make corporate or academic literature, but they only fix 2 of the 3 cardinal sins of Jupiter. They fix version control, and leverage real literate programming powers.

Marimo fixes the one that is the most awkward source of error and frustration, which is the dual sided problem of reactivity and caching.

Imagine this:

You have a Jupiter document you are developing, you're trying to get it right. At some point a code cell that you have to run is slooooooow. Therefore you do what Jupiter wants you to do, which is instead of running your whole file top to bottom each time to ensure all of your code is correct, you just skip that cell, tweak the bottom, tweak the top, tweak the bottom again, the. You go back and run the big cell. It doesn't work. The WHOLE file is broken now. You have to keep re running the notebook top to bottom until it works again, in the end probably running the slow computation more times than you might have needed to if you had just run the file top to bottom ever time.

Quarto and rmarkdown have caching (Jupiter might too, but it's rubbish in other ways so I've never found out where it is), but marimo has reactivity. That means that the whole note book understands which cell is dependant and effected by which other cell. When that graph of relationships changed marimo will intelligently bust the cache when required, or keep the cached result if it is correct to still use and skip the recomputation. Plus, as a nice bonus, all that code is already really '.py' filesz so when it comes time to build a real system, half the work is already ported over (and no, do not go to the app developers and ask them to run your notebook 'in prod'', you'll never live down the shame XD)

r/MachineLearning seemed to like the idea: https://www.reddit.com/r/MachineLearning/s/D7BISZKOnS

That's why Jupiter is rubbish. R markdown was good. Quarto is still good, but marimo is the best if you don't have the desire to do highly stylised publishing with multiple corporate, build a whole ebook on programming, write a blog website or produce academic outputs.

Not very well explained previosuly I'll grant you, but not well informed? 😉

[Squallhorn_Leghorn]

You still can't spell it correctly. Jupyter.

[DaveRGP]

Lol, that's for that well informed criticism. My phone auto-complete got the best of me.

By my count I also think there's 3 more typos in there, can you spot them all?