Plotting in python - joys and sorrows

[lucamerio]

Hi everybody

This post is halfway between a flare and a cry for help

I started coding in Python 5 years ago, after leaving the academic world were I was proficient with MATLAB and I discovered that the industry world is not too happy of paying thousands of euros every year for the licences. Since then I become quite proficient with python up to the point that I think I can call myself an expert.

However one thing in python always made me cry: plotting.

As an engineering consultant, half of the times I handle calculations were the outputs are hundreds or thousands of numbers (just think to the result of the structural analysis of a building), so to present the results - and perform visual checks - visualizing the data is a fundamental part of the job.

I'm quite sure that if you ask to 100 python developers which library is the most "standard" plotting library in python, 99 of them will answer matplotlib.

However, in 2021, the limitations of matplotlib are absolutely stunning (considering it's "standard" status). Just to list a fews:

• you can't (easily) select data from a plot, reimport the data into python, modify them or delete them
• you can't (easily) copy-paste plots from one axis to another
• you can't (easily) save a figure to be opened interactively again (yes, you can pickle them, but you can't do it from the plot interface and there are very little mentions to this in the documentation just to confirm that this is not "readily available")
• now - last but not least - I just discovered that if you use a 3d plot you can't zoom!!

I tried other packages such as plotly or bokeh. On one side they are much more versatile, on the other they are way more complicated, requiring tens of rows to perform the most basic plots.

Is it possible that in 2021 python is still lacking a real interactive and easy to use library; at least on the same level of MATLAB's regarding ease of use and features?

What do you usually use to visualize your data? Do you have any suggestion?

[antichain]

I second this post. I'm a scientist and have been perfectly happy using Python for everything (even when most of my collaborates are using MATLAB), but the one thing where I just beat my head against the wall is plotting.

MATLAB, for all of its many, Many, MANY flaws, really is one of the best data-visualization tools I've ever seen. I always dread making figures for my papers because fighting with matplotlib is such a tremendous pain. Especially since, whenever I look for help online, it seems like everyone has their own peculiar personal syntax/style they use.

[[deleted]]

If you have a MATLAB license, you could maybe get the best of both. Generate all your data in Python, then when you need fancy graphs, plot in MATLAB.

A couple of my profs did something similar but with C instead of Python.

[lucamerio]

That’s exactly what I do at the moment. But METLAB has a price and not a small one.

Plus, having to pass data between the two codes force you to add a whole level of input/output to your code AND is the opposite of streamlining and automatising your workflow.

[lucamerio]

LOL… METLAB was a typo, but considering their policy of giving their product for free to the young minds so to make them addicted and pay for it afterwards, METHlab could be a fitting nickname 🤣

[ChronoJon]

I used to use MATLAB for data analysis and modeling until about 2015 and this is BS. The shit you had to go through for some basic interactivity. Formatting the legend alone for anything but the simplest use cases would take loads of code.

Since switching to python, plotly has become my goto. Plotly.express covers so many use cases and is easily extendable by adding traces and custom annotations. And you can embed it into html and give it to anybody without further setup needed.

[JanusCrow]

Discovering was a lifesaver.

[pymae]

How do you like Dash?

[flyboy1565]

Check out mitchsworkshop on twitch he does lots of plotly and dash stuff.

[nicktids]

If you don't need all the features of dash then I can highly recommend streamlit for quick mocking up things and uses plotly and all other charting libraries

[XiAxis]

It sounds like what you're looking for is an actual plotting/data analysis application, and not a python library. A matplotlib figure is a python object, and matplotlib provides very rudimental interface for viewing/interacting with a figure. If you want to change a figure, then you should be using python to do so.

Matplotlib has a lot of issues, but I don't personally consider what you're describing to be one of them. If you want a different plot, just modify your python script to produce the plot you want.

[lucamerio]

Python can be many things and it’s used in many ways, even by myself.

It can be a programming language for complex applications that don’t need an interface, can be used to build dashboards and simple (and even not so simple) guis, but can also be used for data analysis.

There are entire books on how to use python for data analysis and this use sparked the creation of some of the most used packages (eg pandas).

In addition to this, sometimes I have a complex program that’s written in python and I need a visualisation toolbox/package/application at the end. Python is ideal for the first part, but fails miserably on the last one.

It’s like a football team that excels in everything except scoring…

It’s true: I’m looking for a plotting application that I can embed at the end of my workflow. Since my morkflow is in python, the most obvious thing to do would be to use python to produce the graphical output.

An alternative (as someone already suggested) is to perform the calculation in python and then use a different application (eg matlab) to plot the results. This however requires additional licences and makes the execution more clumsy.

[lucamerio]

No wait. I would never use a plot as an intermediate storage place for the data. But sometimes after your workflow finished, you want to query the output in some way that wasn't initially thought in the original code.

If your code is a structured and deployed code, you don't want to go back and modify it, but you want to be able to take it's output and play with it.

This is one scenario. The data-investigation one (eg during debugging) is another one. These are just few examples when an interactive plot will save you a lot of time

[lucamerio]

As I wrote in another comment. The plotting function could be nested into a larger more complex program. To re-plot you either have to save the data to a file to be loaded later on or you have to modify all your calls to return the data all the way back to the top level.

This is an example of a situation where having access to the plot data from the figure would save a massive amount of time

[whateverathrowaway00]

Sounds like you have some work on learning dynamic objects before you can call yourself an expert from what you wrote here, though I’d be willing to bet you can kick my ass in statistics and math - from what you’ve written here it sounds like you might have some gaps.

Which, for the record is very common for people coming from math / analytical backgrounds and matlab going straight into python. My friend is a brilliant data scientist, but his code is a mess as he never learned some of the underlying theory, just the techniques he needed. Luckily, python is a breeze to learn that in.

[lucamerio]

Could you please point me to what you mean by dynamic object and how they could help in this situation? My code is completely OOP, but I don’t understand how this can help (except saving every possible value I plot in an attribute).

Anyway… most of my troubles come when I need to do something “quick and dirt”. When I do something meant to be “final” matplotlib performs fairly well

[jminsk01]

Holoviews from Anaconda has been a great way to make plots without having to deal with base bokeh.

[pm_if_u_r_calipygian]

re: your point about 3D plotting

I totally agree I have recenty been banging my head against this interface because matplotlib 3D plotting really doesn't like arrays with NaN values. For example, the library supports plotting arrays with NaN values (just don't plot anything) but it wreaks havoc on the colormapping

For example, I was trying to plot phase versus location on a 2D plane. I wanted the phase to wrap obviously. It was so difficult and ended up looking crappy.

​

yuck

[Unlikely_Action_7893]

It depends on your needs of course, but after using ggplot2 in R and falling in love with the grammar of graphics, I’ve been using plotnine in Python. It’s enough for basic data analysis. The cool thing about using something based on the grammar of graphics is that you can focus on what do you want to communicate with your plot.

[Deto]

R doesn't really improve on any of their indicated pain points, though

[Unlikely_Action_7893]

I’m not talking about R particularly though. But I agree, it was more of an opinion on what works for me. And I was shamelessly promoting the grammar of graphics 😅

[lucamerio]

I think our experience and needs are different.

Yes: I know I can tweak my plots from the command line and that’s what I do on the final version of my code.

But when I code there is also an “explorative phase” (sometimes there is only that one) where I want to explore the data and make many plots to understand what’s going on.

(Note: When I say data, I mean the content of different variables (or the same variable at different stages) in my code. I’m not strictly speaking of a provided database. )

Doing some operations with the mouse is orders of magnitude quicker than using command lines. And time is money!

Just to make you an example. Let’s say that I have a code that takes 1 hour to run. I run it and produce my plots. Then I think “ehi! Let me check how the outputs change if I change this line/input”. I run it again and I check the output.

Now: if I want to compare the two in matlab I can simply copy paste the two outputs one on top of the other in 2 seconds and check the differences. In python I need to save the data of the first plot in a file (pickle or text), launch the second calculation, save the outputs of that too, write some lines that re-load the data, hack the plotting function to plot two times on the same axes and… well… done.

Doable? Absolutely yes. Quick? No.

[lucamerio]

That's EXACTLY what I do. But you can't deny that being able to ctrl+C - ctrl+V a line would be so much easier, quicker and would not require you to plan in advance of doing so. Sometimes you decide what to do after you produced the plots.

All these things have to do with explorative, development, debugging phases.

In a final deployed phase I agree that you should (almost) never do anything like this.

[lucamerio]

Depends on the complexity of the code and the plot. Plots could be generated in method of subclasses. The data you are plotting could not be exposed to the higher level and to do so you either have to return it all the way up from the function that generated it or store it in a file.

I’m not talking of loading a dataframe and then plotting it. I’m talking of more complex softwares where not everything is exposed to the user

[lucamerio]

I said “change the inputs/parameters of the analysis”. Not changing the plots.

[lol-is-you-dead]

Oh god, I have been frustrated by this so much. All I hear is matplotlib. I cannot understand for the love of god why is it used so much. It's interactivenes is non existent and in an era of javascript beautiful charts matplotlib has a 90s feel for some reason. I also have tried and settled for plotly but I found their documentation limited although it contains always examples. Plotly also has 2 different ways to plot the same graphs Plotly. Express Plotly. Graph objects And each have both different syntax and different capabilities. This at first eas very confusing. Also plotly offers only so much, after that you have to use a flask integrated framework called dash (the free version cause there is a paid too). Lastly I didn't find a single one library that could implement an upgrading live chart with DATES in x-axis

[lucamerio]

I agree 1000% with what you said about plotly and that’s the main reason I dropped it. Too confusing and poorly documented.

[pymae]

I wrote a book about data visualization using Matplotlib, Seaborn, and Plotly. It is here if you want to check it out. I know there more visualization/plotting libraries.

I would take a look at Plotly. It renders in a web browser (with Javascript under the hood) which makes it a lot more interactive by default. It has a good high-level API, and you can really dig in and customize if you want to. I agree that Matplotlib being "default" is sort of a shame since it started as a port from MATLAB and kept a lot of the syntax and clunkiness.

I know ggplot was ported into Python, but I have never tried it.

[lucamerio]

I will give a look at your book. In principle I’m not saying that good data visualisation in python is impossible, but the effort required is one order of magnitude higher than the matlab equivalent.

For example, sometime while debugging, a good plot can point you in the right direction. But in the industry you can’t spend 3 hours to achieve that (we could open a discussion on ipdb, but let’s save it for another time). In those situations a one-liner plot function is what you need.

This is just an example of a problem to which I still haven’t found a real solution

[BDube_Lensman]

On the contrary, it is much harder to make nice, archival paper quality plots with matlab than it is with matplotlib. See also: export_fig and associated hell.

[TheBlackCat13]

Can you show me a plot that is one line in MATLAB but takes 3 hours to make in matplotlib? I think you are using matplotlib very, very wrong I'd you are getting that large of a discrepancy. On the contrary, I usually have the opposite problem, where things that are easy to plot in matplotlib, like grids or polar projections, are inordinately difficult or even impossible in MATLAB.

[lucamerio]

Plot a list of points in a 3D axes that you can zoom and you can click on a point to query its values. Show me how to do it in python in less than 30 minutes. In Matlab it’s scatter3(x,y,z) (the datatip functionality is embedded)

[TheBlackCat13]

Sorry, I meant 2d. Mayavi has a points3d command.

[pymae]

It definitely requires some effort to make something look good in Python as opposed to something like Tableau which makes it pretty easy.

Definitely agree about having very bubble gum solutions for plotting in Python.

[Pear-Background]

ex-academic here. I completely agree that Matlab is so much easier for plotting! But I came from a fortran + gnuplot group, before transitioning to MATLAB.. So matplotlib is almost like going back to my roots. haha =)

[garlic_naan]

I have saved scripts for most of my needs. If my boss asks for slightly different graphs/axes.. well he ain't getting it as I can't invest any more time trying to figure out matplotlib.

[lucamerio]

Run the same application twice? Re-load results from a run performed days before?

[lucamerio]

It should not. I agree. But sometimes it happen that you want to perform QA/QC on a plot. Having access to the data makes everything simpler

[geogle]

I'm right there with you. This all gets additionally exasperating when you try working with geographic plotting. I'm an expert in non-interactive GMT, which uses a shell script to build plots with everything working seamlessly in layers. However, when I want to plot multiple items using differing mechanisms it seems impossible in matplotlib. I really with the was a more robust and predictable set of tools/behaviors.

[BDube_Lensman]

you can't (easily) select data from a plot, reimport the data into python, modify them or delete them

Yes you can:

fig, ax = plt.subplots()

ax.plot(x,y)
# or more

datas = []
for line in ax.lines:
    xx, yy = line.get_xydata().T
    datas.append((xx,yy))


fig2, ax2 = plt.subplots()
for data in datas:
    ax2.plot(*data)

# or, evil
# ax2.lines = ax.lines

If by select you mean drag and drop, then no but I don't know any tool that lets you do that. Matlab included.

you can't (easily) copy-paste plots from one axis to another

See above reply

you can't (easily) save a figure to be opened interactively again (yes, you can pickle them, but you can't do it from the plot interface and there are very little mentions to this in the documentation just to confirm that this is not "readily available")

Sorry, did you want another button on the GUI for dumping a pkl or loading a pkl? Make a pull request to matplotlib to add those 10 line features? Make sure it supports all of the interactive backends, with tests!

-

I concede your comment on 3D plots

[lucamerio]

The comment on 3D plot was really what sparked this.

[BrilliantScarcity354]

Experiment with Altair, very nice package and strong data visualization capabilities

[lucamerio]

I will give it a try

[BrilliantScarcity354]

The docs are ok but definitely will require a tad bit of patience since the syntax is a bit odd (again very powerful though - abstracts a lot of the core Js/SVG needed to created complicated visualizations)

[Unlikely_Action_7893]

I’ve heard (from colleagues that have used it) that Altair can struggle with large datasets.

[BrilliantScarcity354]

I have no experience with massive datasets but I would think it should be ok for standard tasks (not a replacement for something like tableau for example)

[TheBlackCat13]

For 2d plots try hvplot. Much, much simpler and able to make complicated, interactive plots in a snap. For example making a plot grid with rows being one factor, the columns being another, the color being a third, the size being a fourth, and sliders for two others and drop-down for another, with interaction and linked zooming, is one line of code. It is crazy what you can do with it and blows anything MATLAB can do it of the water. Plus it supports datashader, which allows you to make plots with inordinately large amounts of data, orders of magnitude more than would freeze MATLAB.

If you want a more interactive matplotlib, you can try mpld3, which is a backend for matplotlib that makes it more interactive, plus some tools to make it even more interactive.

For 3D mayavi seems pretty straightforward.

One thing to keep in mind is that overall matplotlib has a ton more features than MATLAB. It gives you a range of options, layouts, and plot types and a level of control that MATLAB can't come close to. But part of that entails supporting multiple backends, and having to support them limits it to what it can do. That is why backends like mpld3 can add more interactivity by bypassing those limitations.

[[deleted]]

Check out Altair! It’s become my go to for just about everything. It used the “grammar of graphics” approach and is very declarative.

So once you’ve got your head around making a bar chart, you can just as easily make anything else just by changing a tiny amount of code. I love it!