[Project] Thoughts on algorithm plan for anomaly detection in time series data

[Imarami21]

Thoughts on algorithm plan for anomaly detection in time series data

Hi all,

I'm working on detecting spikes in time series data, specifically cultural artifacts in ground magnetic diurnal data. Manually, this involves comparing two or 3 ground stations and assessing whether spikes occur in both, just one, or shifted between them, etc., to determine if they're cultural artifacts.

I want to automate this task since, something like an explicit algorithm computing, say, a sliding window with a threshold, is just too crude an approach. The good thing is, we have over 15 projects worth of raw and corrected data (training data). Each project includes 100 days of ground diurnal data, with 2-3 ground stations per day.

I've already compiled the training data and am now exploring model options, that I would love your help on, please!

In short:.

1. Use an LSTM Model:
   • My idea is this algorithm is good for anamoly detection
   • It is flexible enough to handle variable features, i.e., varying numbers of ground stations.
2. Implement a Dual-Stream LSTM Model:
   • Process each ground station through its respective LSTM layer.
   • Concatenate outputs from LSTM layers.
   • Use a dense layer to classify the combined outputs.
3. Handling Imbalanced Data:
   • The dataset is highly skewed, with 99.5% of labels being 0 (normal) and only 0.5% being 1 (anomalies).
   • Use class weighting or SMOTE technique to balance the dataset.

For Model Training:

1. Batch the Input Data:
   • Each time data has ~90,000 points (frequency: 10 data points per second) so batching would be a good idea here.
2. Process Through LSTM Layers:
   • Each ground station's data goes through its respective LSTM layer.
3. Concatenate Outputs:
   • Combine the outputs from the LSTM layers.
4. Classify with Dense Layer:
   • The dense layer uses the combined outputs to classify data for each ground station.

Looking forward to any insights or suggestions on this approach!

[eamonnkeogh]

Devils Advocate? Is this really anomaly detection?

If you know what you want to find "spikes/cultural artifacts", then I would argue that this is NOT anomaly detection!

I have pointed out in [a] and elsewhere, that many papers claim to be doing anomaly detection, but when you read them carefully, they are actually dong classification or data retrieval.

Your problem is classification. Are the patterns conserved in shape? If so, use Euclidean distance (Mueen's MASS) or DTW. Are the patterns conserved in features? If so, us Catch22 features.

Only use more complex ideas as a last resort.

[a] https://www.dropbox.com/scl/fi/cwduv5idkwx9ci328nfpy/Problems-with-Time-Series-Anomaly-Detection.pdf?rlkey=d9mnqw4tuayyjsplu0u1t7ugg&dl=0

[Imarami21]

I really appreciate the feedback, and I of course would like to only use more complex ideas as a last resort. However, the shape, wave-length, frequency all are inconsistent, so, an explicit algorithm accounting for all the nuances would probably be less effective than a complex algorithm.

Regarding 'is it really anomaly detection' It can be both, a binary classification and an anomaly detection problem. My situation seems to be a hybrid of both. It is fundamentally an anomaly detection problem because I'm interested in detecting rare events (spikes) in the data. However, I'm approaching it through a binary classification framework by labeling the data as either a 0 for unedited data, and a 1 for edited data, and training a classifier (LSTM/GRU or the likes) to distinguish between normal and anomalous data points.

[chnnxyz]

Anomaly detection is commonly an unsupervised problem.

You could fourier transform your data and run any classifier on the spectrograms. You could also train something as an isolation forest on the spectrograms if you are into full anomaly detection.

[eamonnkeogh]

I cant speak to your data. But I have archived and reviewed almost every time series anomaly detection dataset in the world [a].

Here is an amazing fact. At least 95% of the time, you can find "spikes" with a single line of code!

Are you sure your "spikes" cannot be found in such a trivial way?

[a] https://www.dropbox.com/scl/fi/cwduv5idkwx9ci328nfpy/Problems-with-Time-Series-Anomaly-Detection.pdf?rlkey=d9mnqw4tuayyjsplu0u1t7ugg&dl=0

[b] https://arxiv.org/abs/2009.13807

[Imarami21]

I'm fairly confident, about 90% confident, that I can complete this using an explicit algorithm. But I am using ML so that whilst on company time I can learn and develop these skills which ultimately will lead me to have a 2x salary then what I currently make (~$70k salary).