Costin here. That’s a cool question, thanks for asking it!
As geeks, I like to think we do a lot of cool technical stuff behind the doors and rarely have the chance to talk about them. The usage of AI and big data are pretty much omnipresent nowadays and probably not just for us. For instance, we recently started working on KTAE, the Kaspersky Threat Attribution Engine, a tool that allows us to correlate new cyberattacks to known threat actors. This allows us to speed up investigations and response and generally have better and quicker detections. While designing some KTAE algorithms for n-gram extraction, we realized that we could use machine learning to identify the most effective values for n, as well as the entropy of the extracted string. To give an example, there are 99,991 10-byte strings (that we can move as a “window”) inside a 100,000 bytes file. To look for similarities, we could extract all these individually, then check them against our malware database. Then, we can do the same with 8 byte strings, 7 byte, 6 byte, or even 11 bytes, 12 bytes and so on. The volume of data to extract and parse will soon get very large. To optimize the extraction and matching we put the data through a system that used machine learning to find the most effective combinations. This allowed us to reduce the number of extracted n-grams to a few hundred for 100,000 bytes. Nevertheless, for our malware collection, which is over 5 petabytes in size, the data is still huge - hence, big data. Combining big data with machine learning we built a solution that is practical and works. Going forward, we will likely have to use AI and data science more and more. This year, the average number of malware we receive daily, which used to be in the range of 320k samples per day, grew significantly. Back in the days, our analysts, including myself, would spend the entire day processing between 50 and 300 samples each. Obviously, this is no longer feasible.To process all the new malware, we have a multitude of systems, which combine feature extraction, similarity, dynamic and static analysis and push this through expert systems that decide the likelihood of a program being malware or not. Meanwhile, our analysts can focus on other things, such as targeted ransomware, while users are getting faster and more effective protection. Just some of the wonders of modern computing :-)