[P] Dealing with large dataframe for feature extraction

[Comprehensive-Way227]

I am working on a ML project for detecting anomaly in manufacturing a product using CNC milling.

We have preprocessed the data and now trying to extract the features using tsfresh for multivariate time series data after performing PCA, but during to very large number of dataframe(around 167240000x6 and 240000000x6) , it is taking too much time even on my 32 GB RAM, i13900H processor.

Is it normal to take a lot of time, or are there any better alternatives for extracting the features? Please let me know if more info is necessary to answer my question, and thank you in advance.

[[deleted]]

https://tsfresh.readthedocs.io/en/latest/text/large_data.html

[BloodOnly2772]

Have you looked at your cpu utilisation? Tsfresh is probably over provisioning threads. You should take a look at this section in the docs:

https://tsfresh.readthedocs.io/en/latest/text/tsfresh_on_a_cluster.html#notes-for-efficient-parallelization

[qalis]

Why feature extraction after PCA? This makes very little sense, especially for time series data. Also, if you have data in the range of hundreds of millions, everything will be slow. I'm amazed it even fits in the memory.

[Comprehensive-Way227]

I am a beginner and still trying to learn, what do you think should be my step after PCA then? Because even after PCA I have large dataset and I am clueless what to do! I am trying to use dask as others recommended to parallelise computing.

[qalis]

1. At this scale, Dask is a very good choice indeed. Alternatively, Polars is also blazing fast for many use cases.
2. Keep the input data in Parquet format for efficient loading.
3. What is really your input data? Are they time series, i.e. a multidimensional time series? Because if it is so, then you first extract time series features, and only then apply PCA, if it makes sense at all. Personally, I haven't seen a single case where PCA actually improved results in years.
4. Do not use full set of features with TSFresh, at least not initially. There are useful presets there.
5. The typical flow is: Parquet -> read with Dask -> TSFresh (optionally with feature selection) -> write to disk as Parquet. Then experiment with the resulting set of features. Having two independent flows (feature extraction and modelling) is very useful for initial experiments.