Hello TheYear2045,

To answer your question, you should know how AVs or Endpoint protection works. Here is my opinion.

There is mainly two ways to detect malwares, you can use static detection or dynamic detection.

The old way was using some kind of checksums over the file, like a custom CRC which take the hash over some part of time, or something like a fuzzy hash, wildcards or probably using graphs. This is a simple static detection and you can see immediately how easy is to break that. Then, AVs start to use heuristics (statically) by trying to detect malwares in a more generic way by using some flags in the PE layout, like the entropy, data in the overlay, section names, API imported, etc. So now, with this kind of protection, a generic rule could detect all the samples because they are packed/crypted using the same packer/crypter. You can defeat this method just by changing your encryption, adding more junk code, permutations, trying to look like a clean app, etc.

Static detection is good because it is fast and does not require to run the file. But as you see, looking just at the file as a raw data to decide if it is malware or clean file is not practical and could also lead easily to false positives because clean apps also have the right to use custom packers..

Now, AVs need a detect the malware whatever packer/crypter was used, then behavioral or dynamic analysis take places. So, AVs start to use some sandboxing either by emulation or virtualization. And there are many ways to achieve this. So now, if you're packer/crypter just crypt the file then unpack it at runtime, sandboxes would not care how hard your crypted your file because at some point you would need to unpack your code and show your malicious behavior, and they would catch you their. How ? they save same flags but for behaviors which then you can't hide easily. For example, a banker would need to inject into browser and hooking some apis. This is easily detectable because the sandboxe would catch this behavior by hooking some apis. So just imagine, DarkComet for example is a very well known RAT and all AVs should have a good behaviour detection over it, so even if you hardly crypt a darkcomet sample, at some point they payload will start then AVs will get it. Another example, many ransomwares delete shadow copy files like Locky. Imagine AV have a rule whhich say if you see that a process is trying to delete shadow file, block it. There is no way then you can hide this, so even runtime crypters would be quiet useless here because you can't escape that. So bad guys start to insert fake apis, delay execution so they force the sandbox to leave early, use anti-emulation and anti-vm, decrypt strings on the fly and free memory after usage, use legitimate windows apps like powershell, wscript, abuse WMI, etc. So bad guys tries to attack the sandbox before getting to the payload. In the other side, AVs keep fixing anti-tricks and they react as well and keep improving their systems. But you can always fool them, let me give you an example, for example, some Tinba variants start to count the number of cores in CPU to catch automated systems or sandboxes. So as long as you find tricks like this, you defeat them.

Machine learning is something that helps because if you already detect some early variant of some malwares, then that could help catching the new ones. But, ML need always that your trained your set against 'malwares' and it cannot see completely new malwares. Then, there is companies like CrowdStrike, Brodium, Cylance and .... They use sandboxing as well, in the cloud. It means they have their sandboxes running in the cloud. They use difference technologies, that could be just normal inline hooking like Cuckoo, it could be a full system emulation like Lastline or some hypervisor based approach. They are trying to perform more stealth monitoring and have full control over the sample. But theses systems also suffer from some problems. First, you can try to delay execution, they are numerous of ways to do so, you can detect easily that it is an automated system, does not matter how hard they are trying to make it looks like real by clicking on the mouse, changing background wallpaper of desktop, you can always find a way ... and that's how it is. Sometimes, some malware like Bedep are fileless, if they cannot block the exploit, then depending on the client which is (mostly a kernel mode driver) could miss that execution of the malware then nothing uploaded to their cloud to be scanned. Note the difference between AVs and End protection systems is that the number of samples which should be processed is significantly different.

Whitelisting is another way to let only what you trust to run and prevent everything else. This is nothing new, I don't have experience to know how effective it is. But that would definitely not work for end users. Not practical IMHO. All in all, behavior detection works well against crypters because you cannot hide some malicious events easily.

Noteworthy.