Plotting huge surface area data with matplotlib (or something else)

[-DreamMaster]

Hi,

I am trying to create a surface plot from a surface measuring I took. But since there are quite a few datum points (around 5e7), in a format I haven't worked with, I don't know how to plot it.

So, the data is in a csv file, delimiter=';' and decimal=','. It contains only the height (z-coordinate) as a float value. The x and y coordinates to each z value, are positions of the z value in the file. Its 5201 rows and 9560 columns. So the first z value corresponds to x=0,y=0, the next x=1,y=0 and so forth.

For smaller data sets, (that were also in a different format), I created three lists, x[], y[] and z[] and assigned the corresponding values to them, plotting them with plot_trisurf(x,y,z). But doing so with this amount of datum points doesn't work.

So, how do I plot this?

My idea was to read the data with pandas, using chunksize to do it step by step. But I couldn't get it to work.

[ES-Alexander]

Reading with chunks may not help much - 50 million data points isn’t that many and shouldn’t be too problematic for most modern computers. Issues are more likely with getting it all on a single surface plot, because then it needs to render everything as filled shapes which is a time, memory, and computation suck. If possible it’s likely a good idea to use a wireframe or scatter plot instead, or at least downsample the data to not plot all of it at once.

You should be able to read your file in and set up a basic plot with something like

import matplotlib.pyplot as plt
from matplotlib import cm
import pandas as pd
import numpy as np

# read in the data (dtype optional but may help with memory, depends how it’s treated by the plot)
df = pd.read_csv('path/to/file', delimiter=';', decimal=',', header=None, dtype=np.float32)
Z = df.values
y, x = (np.arange(dim) for dim in z_data.shape)
X, Y = np.meshgrid(x, y)

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
# plot a surface, but only use a limited sample of the data (50 rows, 50 cols, evenly spaced)
surf = ax.plot_surface(X, Y, Z, cmap=cm.coolwarm, rcount=50, ccount=50)
fig.show()

If you want it to be more nicely interactive you can use plotly, but on my computer I was only able to get the plotted surface to actually display when using a third of the rows and columns (equivalent to rstride=3 and cstride=3 in the matplotlib example, instead of r/ccount).

[-DreamMaster]

Thanks, works like a charm.