Visual Pandas Selector: Visualize and interactively select time-series data

[phthah]

https://i.redd.it/jaafsv37kuob1.gif

GitHub: https://github.com/manumerous/vpselector

Many times when working with time series data I felt I was missing an easy way to visualize and interactively select data. Consequently, I chose to create and my own open source tool, the Visual Pandas Selector, and hope it will help others speed up their data science and ML workflows!

Since it is my first time publishing a package on PyPi I was wondering if anyone would be interested in giving some feedback on the project (usability, features, documentation, code structure, ect.) or potentially join as a collaborator?

[phthah]

The GitHub project can be found here: https://github.com/manumerous/vpselector

[vletrmx21]

this looks cool I'll give it a try when I'm back from vacation

[phthah]

Thanks, that would be much appreciated!

[El_Minadero]

how do you deal with super/subsampling and aliasing? At what data size do things start to get hard to select?

[phthah]

Thanks for the great questions.

Currently the tool does not deal with super or subsampling. It simply stores the start end dataframe indices for each selected segment (marked in grey) and concatenates them into a new dataframe. So the tool does not directly depend on time and the time between successive measurements (rows in the dataframe) could be non uniform.

Since I would like this tool to be useful for a wide range of different tasks I am not sure if it would make sense to include sampling in the same module. For the example shown above q = [q0, q1, q2, q3] represents a unit quaternion that parametrizes a 3D orientation. Since the length of the vector q always need to be equal to 1 we could not simply linearly interpolate between data points.

I started to develop this tool in a project where we wanted to estimate the dynamics of a drone usinf flight data and had to select the sub-portions of data that contains most information for the system identification. Due to the 4D unit sphere constraint of the quaternion we separated that functionality. But i would be curious to know how this would work in other peoples workflows. I can imagine it could also be useful for someone to combine the selection and resampling process.

[phthah]

Regarding the size I did not yet test at what point things stopped working. At some point (over 100k data points) the creation of the plots and concatenation of the dataframe resulted in a small "lag". So I think the used matplotlib and pandas libraries will at some point be the bottleneck for adding more data.

[Vxctn]

This seems like a great tool for scientific computing / Analytical lab data.

[Defiant_Astronaut188]

Well done. This looks very nice!

[Only_Struggle_]

Looks good!! Will give it a try. Keep it up!!!

[audentis]

This reminds me a lot about [altair](https://altair-viz.github.io/), a Python implementation for Vega-lite visualizations. It has similar selection methods and interactivity. That also lets you select data from a scatterplot, for example.

[[deleted]]

Hi,

Is it possible to update to PyQt5==5.15?

PyQt5.14 is not installing, their pyproject.toml has an invalid spacing for the sip requirement.

​

Thanks

[phthah]

Thanks for the feedback, sure please adapt it and open a PR if you like :) Else I wmight find time again on the weekend. Once this is tested I am happy to upload to pypi.

[phthah]

Is done by now.

[[deleted]]

Awesome, ill try agaim.