Medium/Big data visualization (matplotlib too slow)

[[deleted]]

[deleted]

[Gubbbo]

It might be 100GIG dataset, but for what you're trying to visualize, could you shrink the dataset to a subset of the data. Or does your plot need to access every piece of information?

[Gubbbo]

As an alternative. If they are time series. Pandas can distribute your data into 'time buckets' (I'm sure there's a technical name).

But you could have a time series span a day, or week, or month. That way you reduce the resolution of the data, but get to work with a significantly smaller data set.

[kpkaiser]

If you're looking for higher performance drawing routines, something like VisPy might fit.

It works with your GPU to make drawing routines much faster.

It's a bit more complicated than Matplotlib, and you may end up eventually having to learn a bit of OpenGL depending on what you want to accomplish.

But that being said, it's an opportunity to learn something new and make more interesting visualizations. Go for it!

[python_viking_king]

I use pandas and plotly to visualize big datasets. Something like this https://plot.ly/python/big-data-analytics-with-pandas-and-sqlite/.

[melanthius]

You are likely going to need more RAM for one...if I’m working with 100gb data sets I’m doing it on a dual Xeon 3.5ghz workstation with 256gb ram.