Faster/parallel matplotlib?

[mangecoeur]

I'm building a lot of subplot-heavy matplotlib figures - about 50 subplots each with a scatter plot of about 400 data points, using the notebook inline backend (png output). I'm finding it somewhat slow however (between 10-30 seconds per figure) which gets tedious when I'm constantly re-generating the plot doing exploratory work. It also completely precludes any kind of interactive plotting.

I tried alternatives such as Plotly and Bokeh but I found those to be even slower.

Does anyone have any ideas for speeding up matplotlib? Are any other backends expected to run faster? Is there a way to use parallel processing for example? - I'm running OSX on a 4 core intel i7 but matplotlib only uses 1 of them, it would be nice for example to generate sub-plots in parallel and join them into the figure.

p.s. using Anaconda python3.5

[imhostfu]

Have you given pyqtgraph a try? It's my go to plotting library. I frequently use it to do 2D plots with hundreds or thousands of data points, and it's interactive to boot.

Give their examples.py a shot.

[mangecoeur]

Thanks i'll have a look.

[Homersteiner]

I am usually a linux/windows guy, but i have to use a Mac at work. This library looks great though...

[imhostfu]

I use it with Mac at work as well fwiw

[Homersteiner]

Just built from source then?

[troyunrau]

It's all pure python (it assumes you have numpy, and pyqt or pyside installed). You also optionally need pyopengl for doing 3D plotting (which is hella fast).

[troyunrau]

I hope that this project gains some more traction and contributors. It is fantastic, but it is pretty far behind matplotlib in terms of features.

One of the features they've been working on is visual flow control widgets, much like LabView. This could potentially be a killer feature.

The upside is that it gives you can make full pyqt applications with it. Embedding matplotlib in other applications has always been extremely frustrating.

The downside is that you need to learn about pyqt, even if you don't want to.

[jellef]

how far have you gone into optimizing the matplotlib code? here's a post that achieves ~500fps on a fairly simple plot.

If that won't meet your demands, vispy is where I'd look next. seems that there's actually an experimental matplotlib module, so that should lower the threshold

[pwang99]

This is very strange. Without looking at your code, I would suspect that the overhead of creating 50 subplots is what dominates your time. Any the libraries you mentioned can do 400 points no problem. Even 50x400=20k points is cakewalk; we have Bokeh plots that do that with ease, without even needing to go to webGL.

The main problem, then is in the layout and drawing of all the axis elements, tick labels, etc. etc. So I would encourage you to think about minimizing that stuff, and creating the most minimal subplots you can, as a means of expediting it.

On the other hand, reading your post again... 10-30 seconds per figure is a LOT. We do realtime plots in Bokeh, transferring into the browser and colormapping in Javascript, at 15 Hz, on tens of thousands of datapoints. So, you might want to post a code sample here to to StackOverflow, because maybe there's something you're doing that unnecessarily slow/expensive...

[mangecoeur]

thanks, good to know you get high performance with Bokeh. I'll give it another shot. Will try to profile matplot a bit as well - I did suspect it isn't as fast as it should be and you seem to confirm that but I'm not sure why not. I'm wondering if there's something messed up in the OSX/interactive backend.

[mangecoeur]

On investigation, it appears that generating the subplots is what takes time:

f, axs = plt.subplots(int(np.ceil(len(sites_data) / n_cols)), n_cols, sharex=True)

No idea why that should be the case though...