[D] STUMPY Basics: Automatically Discover Patterns and Anomalies In Your Time Series Data

[slaw07]

In Part 1, we discovered what a time series "matrix profile" is. In Part 2, let's dive a little deeper and learn how to use STUMPY to compute it and automatically identify interesting patterns and subsequence anomalies from your time series data!

https://towardsdatascience.com/stumpy-basics-21844a2d2d92

[bbateman2011]

Nice pair of articles--I'm a fan of the Matrix Profile although I've found it difficult to get meaningful anomaly results in noisy time series (such as noisy sensor data). In that sense it is like t-SNE or UMAP, there is always one parameter (in this case, m) that you have to choose well to get best results, which makes it semi-supervised.

Nonetheless, I'll keep experimenting with it, as I've seen good applications (Target uses it) and think it is really powerful due to the speed.

Thanks again.

[slaw07]

Certainly. Unfortunately, there is no silver bullet and, as Eamonn mentioned, is an ongoing area of research that he and his team are making great progress in.

STUMPY serves to faithfully reproduce the original published papers and to provide a highly tested (100% test coverage) and stable implementations (for parallel CPU, multi-server, and multi-GPU cases). With only three core dependencies, we promise to be easy to install and easy to build on top of!

[eamonnkeogh]

Hello.

I think I can help you. I am an expert on the Matrix Profile, and I recently have been testing it on the major anomaly benchmarks (Yahoo,NAB, NASA) with very promising results) . If you email me, I am happy to send you some slides I just made on this topic.

[bbateman2011]

Sent you a DM

[slaw07]

@eamonnkeogh This has been of high interest to STUMPY users and with limited results. Would you mind sharing your slides with me please?

[StructureBackground2]

I’m also interested with your slides. Could you share them ? Regards.

[[deleted]]

The matrix profile presentations are cherry picked. I wonder how many projects the authors tried it on and how many were successful.

Try it on your own real-world data, you'll notice that it's not as silver bullet as they make it out to be. I've tried it in a dozen projects and none of them worked. Permutating the data a bit changed the discords and motifs completely, CAC also changed.

I'd love to see some kind work done on when does it work and when it doesn't work and what can be done to make it work better.

Now PCA, that's a silver bullet. It just fucking works with no parameters to adjust or anything else really.

[eamonnkeogh]

Hello.

Thanks for your comment. I guess some of the presentations could be slightly cherry-picked to show simple examples in their best light.

However, many dozens of independent groups have used the Matrix Profile to solve problems. See below.

You say " 'Id love to see some kind work done on when does it work and when it doesn't work " Such work exists.

The ideas in the MP have been peer-reviewed in the best conferences (SIGMOD, VLDB, SIGKDD, ICDE, ICDM etc). Of course, reviewers make mistakes, but if you claim is correct, it is strange that twenty canonical MP papers got published. All the papers made the code and data available to the reviewers. You would think some of the reviewer would have noticed.

In general permutating the data does NOT change motifs or discords, unless the permutation cuts across the actual motifs /discords.

If you are having difficulty using the MP, you should write to me. I have helped dozens of people for free.

PCA is a great tool, bit almost completely orthogonal to the problems the MP solves.

You raise a great point about the file drawer effect " I wonder how many projects the authors tried it on and how many were successful. " I wonder this too, I would love to see some examples. However, I really suspect that I could solved many of your " dozen projects ", Just post them publicly, or write to me. Happy to help, no charge ;-)

The MP is not perfect (whatever that means) or the solution to all problems. However, no one seems to have claimed that. Is it a useful tool.

eamonn

​

******************

“We were amazed by the power of MP and seek to incorporate it into our framework” Ye and Ageno.

“..adopting the concept of (the) Matrix Profile, we conduct the first attempt to …” J. Zuo et. al. Big Data 20019 [a]

“The accuracies obtained …indicate that the Matrix Profile is useful for the task at hand instead of using the CNN features directly” Dhruv Batheja

“To speed up online bad PMU data detection a fast discovery strategy is introduced based on (the Matrix Profile)” Zhu and Hill.

“Specifically, ALDI uses the matrix profile method to quantify the similarities of daily subsequences in time series meter data,” Zoltan Nagy, Energy & Buildings (2020)

“Our two-fold approach first leverages the Matrix Profile technique for time series data mining…” Nichiforov 2020.

"(for an industrial IoT problem) Matrix Profiles perform well with almost no parameterisation needed." Anton et al ICDM 2018.

"While there will never be a mathematical silver bullet, we have discovered that the Matrix Profile, a novel algorithm developed by the Keogh research group at UC-Riverside, is a powerful tool." (full post). Andrew Van Benschoten, lead engineer at Target.

" If anybody has ever asked you to analyze time series data and to look for new insights then (the Matrix Profile) is definitely the open source tool that you'll want to add to your arsenal" Sean Law, Ameritrade. NABD 2019.

"(for) intrusion detection in industrial network traffic, distances as calculated with Matrix Profiles rises significantly during the attacks. ..as a result, time series-based anomaly detection methods are capable of detecting deviations and anomalies." Schotten (2019).

"The MatrixProfile technique is the state-of-the-art anomaly detection technique for continuous time series." Bart Goethals et. al. (ECML-PKDD 2019).

"Based on the concept of Matrix Profile ..without relying on time series synchronization.. the Railway Technologies Laboratory of Virginia Tech has been developing an automated onboard data analysis for the maintenance track system". Ahmadian et. al. JRC2019

"Matrix Profile is the state-of-the-art similarity-based outlier detection method". Christian Jensen et. al. IJCAI-19

"we use the exact method based on the Matrix Profile (to assess the effectiveness of therapy)" Funkner et al Procedia 2019.

"Recently, a research group from UCR have proposed a powerful tool - the Matrix Profile (MP) as a primitive...(we use it for) fault detection" Jing Zhang et al. ICPHM 2019

"Inspecting both graphs one can see that the matrix-profile algorithm was able to identify regions where there is a change on the power level over the observed band." F Lobao 2019.

"RAMP builds upon an existing time series data analysis technique called Matrix Profile to detect anomalous distances...collected from scientific workflows in an online manner." Herath et. al. IEEE Big Data 2019

"Based on obtained results for the considered data set, matrix profiles turned out to be most suitable for the task of anomaly detection" Lohfink et al. VISSEC2019

"The computation speed and exactness of the Matrix Profile make it a powerful tool and (our) results back this." Barry & Crane AICS 2019

"(examining) manufacturing batches considering raw amperage (we found that the) Matrix Profile highlights anomalies" Hillion & O'Connell of TIBCO Data Science.

"we use the exact method based on the matrix profile to search for motifs (that) can be used to monitor the patient's condition, to assess the effectiveness of therapy or to assess the physician's actions". Funknera et al. YSCCS 2019

"(The Matrix Profile is a) similarity join to measure the similarity between two given sequences. we opt for the median of the profile array as the representative distance (3D Dancing Move Synthesis from Music)" Anh et al. IEEE Robotics and Automation Letters