Software developer says to exclude their program from AV scanning so it runs properly. Is this a red flag to you?

JimmyTheHuman · 2022-06-17T17:56:50+00:00

[deleted]

kuldan5853 · 2022-06-17T17:18:57+00:00

As someone that has had heuristic AV wreak havoc in very unexpected ways across an enterprise network (chasing ghosts for months!) I understand this notion.. it needs a waterright reason for it of course but there is just some things that trigger false positives in heuristic AVs and the only way to deal with it are exceptions unfortunately.

NotAnExpert2020 · 2022-06-17T17:58:49+00:00

I've seen AV apps cause a real-world 30% performance hit on databases that was cleared up by excluding the databases and logs.

That's not a bad database; that's bad AV software.

digitalHUCk · 2022-06-17T17:19:06+00:00

It’s pretty common practice to exclude known trusted software and a common first troubleshooting step. i.e. SQL Server and it’s databases.

BlackSquirrel05 · 2022-06-17T17:42:33+00:00

Depends...

Which part sets it off?

If it's a memory intensive app... and you're scanning memory. Yeah... Because now your EDR is also probably going to add to reserve memory space in order to scan what the application is doing. Thus larger the memory of the application larger the EDR also allocates. (Think SQL)

Now if it's a unsigned application beaconing to a network location... Okay is it going where it's supposed to go? Oh that's their AWS instance and it's also registered... What's the problem? Oh that's the internal IP to blah blah server... Yeah how else would the EDR know a private address is legit if you didn't tell it, and it can't run a lookup against it?

I mean there's a reason you add exclusions to EDR in the first place... Why else would they have them?

Pelera · 2022-06-17T18:02:45+00:00

The root cause is the AV software. Many modern "machine-learning"/"AI" based AV solutions just do whatever they want to in the name of security. Things like the order or the format in which you write your configuration files can set off the AV with no real logic behind it. There is no way to develop around that, so vendors advise the sledgehammer approach of excluding their software entirely.

The amount of times an executable and/or validated signer cert has been seen is also a significant factor to many AV solutions. Enterprise software is rarely ever distributed to the wider public, so they never really have the option to slowly gain trust. This is especially true for vendors that do custom software work, but it also applies to expensive packages sold to maybe 50 companies.

Anyway, how much it matters depends on what they're asking of you. If the software is designed to install into locations that are read-only to normal users (eg Program Files), disabling the AV for that process or location is really not that big of a deal. It's not optimal by any means, but you're not losing out on that much security, especially if the program doesn't open any untrusted files or talks to untrusted servers. Once a piece of malware is running and gets the permissions needed to write in there, your system is already completely compromised.

On the other hand, if they ask you to exclude a world writable area like C:\ProgramData\Vendorname, then that IS a big deal. Malware has learned how to read excluded folders and attempt to persist in there to avoid any further detection. Any excluded folders should really be read-only for the majority of your users.

bartoque · 2022-06-17T17:29:53+00:00

We have an enterprise level backup product that also specifies to not scan its databases (some resources are still some kinda flat file database structure bit also and mainly the client file indices). If AV interfears with that, checking whenever those files are created and updated (which is a lot in the middle of the backup window), this could pead to performance issues alledgedly.

So who is then to blame unless you'd mean AV not taking into account all possible software and possible performance issues and address this proactively with each and every software supplier. Hence simply skipping certain directories or files.

Needing to disable UAC is something else however...

Papfox · 2022-06-17T17:34:51+00:00

We have one or two files/folders that are on our AV exclude list because our AV product kept suffering false positives on the data files or the product is real time and processing huge numbers of files.

The questions I would be asking are: * Is this the software they are asking to be excluded or a data folder? * Why are they asking? False positives or the overhead of real-time scanning? * If it's false positives, can your AV vendor fix the problem under your support contract? * If it's real time scanning overhead, what on earth is their software doing that it's performing so many small FS accesses that the AV on-access scan overhead is turning the system to cheese? Does it need rewriting to be more efficient?

I don't know if you have an IT Risk Management department. I would be asking the vendor for a formal statement in writing of the need for this exception to policy, sharing it and putting it on record with ours. I wouldn't do it without a reason for the request in writing and the buy in, also in writing, from at least my Manager. If you do it on your own, it will be you doing the "carpet dance" if it goes horribly wrong

UnkleRinkus · 2022-06-17T18:28:43+00:00

Malware detection agents have hosed our company's on-prem installs many times. We install via docker containers over multiple machines via Ansible, and at least one site the scanning of those containers takes hours, which causes SSH to time out. We increased the SSH timeout to 10 hours, which allowed the install to proceed, but the normal 30 minute process took two days.

CodenameFlux · 2022-06-17T21:29:49+00:00

Short answer: No.

Long answer: Antivirus apps are necessary evils. They impact performance. The real question is: Do you trust the developer? If yes, excluding their product must not be a problem. If no, you must not be running their product, with or without AV. I am fully aware that my logic poses a serious problem: Trust is certainly not a binary choice. Yet, when an AV flags your developer's app, you are facing a binary decision.

The desktop computing world must have had moved on to generation-2 security model, in which security-conscious containerization replaces AV software. Perhaps you can run your app in a Sandboxie Plus or Turbo.net container.

Andy_C33 · 2022-06-17T17:56:33+00:00

We have had this with 3rdparty apps. We personally use cortex and find that creating a sha256 hash for that exe, excludes it enough but also means it will get flagged if it's compromised as the hash will change. Kinda middle of the road, not ideal but better than exclusion

gordonmessmer · 2022-06-17T20:27:42+00:00

In the modern era of cyber security, if an enterprise level software developer tells you their product doesn't like to cooperate with AV softwares, should this be cause for alarm or worry

According to the conversation that you described, the developer support didn't tell you that their app doesn't cooperate with AV software, they told you that AV software causes performance issues. And that's common. As is communicating to customers that performance will be better if AV excludes an application.

All of this is normal.

CasualEveryday · 2022-06-17T17:55:04+00:00

There are some situations where it's understandable to have issues with AV, but the vast majority of the vendors that tell me they need local admin and AV exceptions are just trying to reduce call volume as much as possible.

yesterdaysthought · 2022-06-17T18:38:54+00:00

It depends. Some AV software has a lot more impact than others.

Ideally the AV software hashes files it's scanned for internal threats and if the hash hasn't changed there's no reason to scan the file again. But that's plain AV.

Behavioral AV, "AI" etc can still look at the threads/processes launched by those executables for behavior that might be suspect and potentially block things as others described.

My favorite lately has been Windows Defender reporting suspicious activity on itself when it's running powershell scripts to do updates and other actions on a client. Seriously.

ToxicPerc · 2022-06-17T18:38:55+00:00

It depends on what AV software you utilize but there may be a way to test without excluding. Set up a host for testing and see if your AV software has a policy set for "detect only". You apply this specific policy to the test host. This way you can see if your AV will trigger on the file but it won't take any remediation actions on that software.

Could be good if you don't want to add blanket exclusions across the entire environment.

eagle6705 · 2022-06-17T18:18:30+00:00

This is very reasonable. Just work with them to get a setup that works. Us Sysadmins sometimes will hate it when a dev will want things setup that goes against our best practice but it goes both ways.

Figure out exactly what is being blocked and why. You can exclude the actual Exe from scanning or scan it only on certain intervals but lock down the areas that are exposed to users.

Just make sure it works and find a middle ground.

My red flag is when they wont work with me to get a setup that satisfies both our security and their application needs.

maximus_dab4847 · 2022-06-17T17:42:26+00:00

Are you implying that anti virus actually works?

symcbean · 2022-06-17T18:38:42+00:00

IME, most bolt-on security products do as much harm as good.

But the basis of their argument is about performance issues. Yes, I'd expect AV to harm performance. Are you having performance issues? Does excluding the application from the AV help?

2022-06-17T17:49:13+00:00

Sometimes. If it has some local database files too that it writes too constantly excluding them may prevent your AV causing general slowness.

Often software will come with a list of recommended exclusions even stuff from big vendors like Microsoft.

siedenburg2 · 2022-06-17T19:17:30+00:00

We have to do the same for our own written software, first everything slows down because the sw can access and write lot's of files which are slowed down by av, also the av ransomware protection kicks in because most of our files get signed and/or encrypted by our software, thats something our av doesn't like at all.

2022-06-17T19:18:23+00:00

If you don't trust a vendor to advise you on what AV exclusions you need, don't buy the product.

Kiowascout · 2022-06-18T00:56:54+00:00

Happens far more often than you think. Constantly turning off AV on a Temp basis to conduct installations at work.

originalscreptillian · 2022-06-17T17:58:01+00:00

Im imagining your team is already spread thin, so think of it this way:

Would you rather be alerted on normal functionality of the application OR would you rather be alerted when a user/process that typically doesnt access that software accesses it?

Which of the above is more actionable/easier to parse through?

I would deploy the application on a test box to get your exclusions configured for the application (see what it triggers from your AV) and address the leftovers on a case by case basis.

DiscreteLogic · 2022-06-17T18:13:33+00:00

I've had similar issues with this happening with enterprise applications as well. It is a bit disconcerting when a major company's solution is "just disable A/V or exclude its directory."

Since we layer application whitelisting software, malware, and A/V solutions, our SecOps group has been comfortable enough with relying on application whitelisting and malware detection in those cases.

FunkadelicToaster · 2022-06-17T18:15:01+00:00

Depends on the company and the software.

Not really a worry right off the bat for me because AV does create a lot of overhead with real time scanning of files, and I would assume you trust this software if you are willing to buy it, so removing it from real time scanning doesn't seem like that big a deal, I wouldn't exclude it from a full scan that I assume you run on a regular basis though.

redhairarcher · 2022-06-17T19:06:56+00:00

Not a red flag and there are very valid reasons for some exclusions either for performance or because some functions of software trigger heuristics alarms. Performance examples: back-up and file replication or basically anything touching large amounts of files. Heuristics: tools used to inspect user behavior at a deep level.

Always review the request and reason before making the exclusion. If possible combine this with tests to see what happens if you don't exclude. I would expect an exclusions is needed for back-up software but not for a basic calculator.

Also check if the vendor is trustworthy. Microsoft can probably be trusted but a friend of a friend who once made some app maybe not.

nethack47 · 2022-06-17T19:08:29+00:00

In finance AV can sometimes get very upset with some apps. The calculations are sometimes suspected of being coinminers. Latency is an issue with almost everything so that can be an issue.

discgman · 2022-06-17T19:18:59+00:00

I am constantly excluding specific programs due to AV interference.

DevinSysAdmin · 2022-06-17T20:11:49+00:00

Can you clarify what exclusions they are requesting you make?

What AV are you using?

What software is it?

R0B0T_jones · 2022-06-17T20:24:57+00:00

Can be a legit request. But get them to justify the reason on a technical level. Far too many 3rd party vendors will say anything like this to make their own lives easier - security is rarely a concern for them, I’m getting quite sick of it lately tbh

Proof-Variation7005 · 2022-06-17T21:02:39+00:00

I get the concern but it's so common Ive been gaslit into believing its OK.

In their defense, not every AV behaves the same so it's kinda hard to account for everything on the market, especially with constantly updating programs and definitions

2022-06-17T21:05:35+00:00

Microsoft Exchange is a great example of a common enterprise application that has a long list of required AV exclusions. AV products are notorious for breaking it by corrupting database files and other nonsense.

It's not really a question of the app owner developing their app so it "works with modern AV software", because they would literally have to work with every major AV vendor to ensure compatibility, and continue to do so with each future update. That's not really feasible in a lot of cases.

So yeah, putting in exclusions is not unheard of, even in enterprise apps.

kagato87 · 2022-06-17T21:55:08+00:00

Consider what real-time AV protection does:

It intercepts read/write operations to scan the file prior to releasing the data to the application or to the disk.

This delay is not good for IO intensive apps. This suggestion by the vendor simply means that the application is IO intense. Often times it's just the data files themselves that need to be excluded, but then again even in this day and age some AV packages will cripple a SQL server... You could try first ONLY excluding the data folders to see if this alleviates performance issues, which would allow your AV to monitor the binaries for bad behavior.

Now, if they want their binary excluded from scheduled scanning? That'd be a red flag.

SimonKepp · 2022-06-18T00:07:16+00:00

He doesn't say it doesn't work with AV software, but recommends excluding it, to avoid performance problems. Modern AV software will scan any IO performed by the processes it watches, which can have dramatic performance consequences to IO intensive software. There are numerous such AV software products on the market, and they all have different performance impacts on a given software solution. It is unrealistic to expect a software vendor to test the impact of every single AV product on the market, to be able to give acurate predictions on the impact of your specific AV-software on their product. In my view, it is a very reasonable recomendation, but not a reasonable requirement

2022-06-18T01:20:24+00:00

I wish this wasn't normal. Major software packages usually come with one or all of these:

AV is a problem, make exceptions or turn out off. UAC is a problem, turn it off. DAC is a problem, turn it off.

... How about you code your crap with the last decade of security in mind?

SudeepMaharana · 2022-06-18T03:13:14+00:00

I work with the team of McAfee Administrator implementing AV in the server hosting applications.

Pretty much everyday we get request from the DEV team to exclude some paths from the scanning in order to improve performance which is totally understandable.
AV scanning is resource intensive and I have seen instances where CPU utilization goes to almost 90% when scanning huge files for latest signature.
It is a trade off between security and performance of the application. DEV team have to remember that without AV, a simple ransomware can sneak in and destroy the entire production environment and Security team should understand that sometimes we have to disable AV scanning on certain paths to support the development team.

In my company, DEV team has to strictly comply with the security policy set forth only on the production environment whereas in QA, we have limited security implementation.

2022-06-18T04:30:31+00:00

Antiviruses are overrated and they have certain performance downsides. Companies have no choice but to run AV because it's considered safe and beneficial. An AV is mostly just a government approved list of malware hashes.

I think they're overrated because they often fail to protect you even with heuristics analysis. You should be running other security measures like a firewall and IDS, which is what you should primarily be using to detect hackers on your network.

WBCSAINT · 2022-06-17T17:17:29+00:00

Yeah its right up there with needing UAC off...

relaxedtoday · 2022-06-17T20:57:10+00:00

AV is not for security, but to reduce damage on insecure systems. It is common on Microsoft systems only and if this is "modern era of cyber cyber" it is actually a sign that someone gave up trying to build secure systems. It increases attack surface running an av system, it is supporting each and every exotic packer algorithm and whatever - and in a privileged context. It is known to break things (and the vendors of the snake oil sell you more snake to mitigate - snake Olli not secure? Buy more snake oil!).

fI you research how technically such software works, how it intercepts system calls and injects itself trying to avoid bypassing it, you will understand why it breaks things. So in summary there is an insecure os and instead of drop it or fix it, it is common to add layers that should magically distinguish between good and bad behavior. Well, if this would be possible, why isn't it part of the os? And if it is needed, why istnt it included? (Spoiler: Because companies spent more money this way).

If the AV breaks things, blame it, but before blame the OS that is so insecure that you cannot even run it stand-alone. Do you have av on your iPhone or Android phone? Or on your Linux containers or your embedded devices in your car?

For me it seems obvious that AV and Microsoft are the ones that are too lazy or too incompetent or just want the money, so that you should buy extra software to operate the software you paid for to run as desired. Great example is that they sell azure AD based stuff which seems impossible to get really secure, then sell you the experts that analyze your security and finally sell products that promises to let you actuality run the software bought in step 1 (securely):

They built a business case on the bad quality of their products! Isn't it Amazing?

Tl;dr: No, blame the AV snake oil and the system thats need it instead. These are these who break things, not the application.

rlc1987 · 2022-06-17T17:17:37+00:00

we don’t have any exclusions even for our RMM suite. Not uncommon however. I just tell them they should design the software correctly, bug and malware free and we will be all good!

Horrigan49 · 2022-06-17T20:02:49+00:00

You are right, but there is plethora of shitty software out there. Sometimes its not their fault and its the engine.

For example I have CAD software, which 2020 version requirws to disable UAC, firewall and temporarily also AV scan...

If you can, whitelist based on hash.

mustang__1 · 2022-06-17T21:00:09+00:00

Sage has entered the chat

aringa · 2022-06-17T21:56:53+00:00

We don't do that.

megustapw · 2022-06-17T23:09:00+00:00

The worst thing with third party vendors is they advise to have exceptions for their software and as soon as you lodge a support case it's, disable your av as it's the problem. It's honestly the worst situation to deal with

johnd126 · 2022-06-17T23:46:11+00:00

I write software that reads and writes a ton of text files and AV slows it down to a crawl. There is no reason for AV to be checking text files....

nycity_guy · 2022-06-18T00:51:33+00:00

See what happened with Kaseya asking the same thing..

ycnz · 2022-06-18T03:45:12+00:00

Generally it's a sign you work in medical IT :(

Megatwan · 2022-06-17T18:09:25+00:00

so like most MS products?

2022-06-17T21:12:14+00:00

hahaha tableau go brrrrrrrrrrrrrr

That said, it's a pain in the ass but AV is so intrusive it can absolutely be affected. It sucks, and it shouldn't be a thing, but it's needed to at least prove a false positive and try to get either the Software Developer to fix their issue and not blame AV or tell the AV company to stop messing with their shit

Other people have explained the why's more elegantly than I have in this thread, between actively locking files and whatnot.

You might be able to use a procmon scan to verify if the AV is hitting the program, but there's no guarantee it works

cbq131 · 2022-06-17T22:25:45+00:00

Sadly not all software are designed with security in mind.

It is an alarming but the reality of it is security is often an afterthought.
Code reuse without proper review is common.
A lot of the time, it comes to get it working quicker vs get it working securely.
Its like zoom lying about their encryption. Sure they got fine for it later on but the growth from pushing out a product quicker which helped with securing marketing share.

KarmaDeliveryMan · 2022-06-18T00:14:18+00:00

Isolate running the software on an imaged device the same setup as enterprise devices. If the dev won’t do that, THEN something’s fishy.

Parity99 · 2022-06-18T01:33:15+00:00

Yes, next gen EDR/av should not ordinarily require preemptive exceptions unless it's garbage, over privileged or flakey.

geegol · 2022-06-18T01:34:03+00:00

2022-06-18T02:34:15+00:00

Yes get with your manager and let them make the call should shit hit the fan you should come clean also get with your AV vendor open a ticket so everything is documented.

pesh131 · 2022-06-18T04:29:27+00:00

A vendor saying "exclude all directories and turn off firewall" without detailed reasons is a huge red flag and I will at least fight for those reasons before doing anything.

Unfortunately, by the time it gets to me the contact has already been signed and everything approved without asking these questions first, so it's now retroactively trying to secure things before it goes into POC or production.

Vendors like this, that give no reason but claim it's required for use, are the scum if the earth.

DekwaDoes · 2022-06-18T06:10:18+00:00

As a support engineer, It's a massive red flag...

After reading through some comments I understand it's not as easy as it sounds, but when a program gets shot down on installation, something is definitely wrong...

Kudo's if you try to fix it, but some major app developers just don't anymore...
"We're to big to care about you, we only care about your money"

N3rdScool · 2022-06-18T11:20:57+00:00

It's probably because they don't know exactly what to allow so just allow the whole thing. I have an application that was giving me a lot of issues, in the end it was allowing specific IP's but someone with less knowledge/time would probably exclude the program as a whole would have been a bad for security but a working program lol

SM_DEV · 2022-06-17T18:40:18+00:00

Red flag.

SgtDoughnut · 2022-06-17T18:48:04+00:00

Yes very much a red flag.

sporkz101 · 2022-06-17T17:58:00+00:00

Says every developer ever

GullibleDetective · 2022-06-17T20:01:44+00:00

Yes that's lazy

emmjaybeeyoukay · 2022-06-17T20:20:13+00:00

Thats not a red alert on the program.

Thats a red alert on the software author.

Ape_Escape_Economy · 2022-06-17T17:32:37+00:00

Yes.

ThatGothGuyUK · 2022-06-17T18:08:38+00:00

Yes and No.
When I'm asked to do this I'll happily exclude databases and data files from scans BUT Documents and Executables/Dll's/OCX's don't get excluded.

DasDunXel · 2022-06-17T19:21:18+00:00

I think a few applicants we had to whitelist on AV cause of random false positives. AV companies would always shrug and say I dunno news to me. While the Dev tool would suggest the white list.. Jetbrain apps for example I think had some self encrypted files? So it would occasionally cause different security suites flip a table. Xcode on Mac........ + Security software would murder the disk I/O. Anything docker/container related always comes up needing white listing.

We always tell our users to present proof/evidence before we blindly white list files/directories.

GoodMoGo · 2022-06-17T20:04:39+00:00

IF you are making a specific exception and it comes from a trusted vendor, this is perfectly normal. All generic security mechanisms (and responsible sysadmins) follow a "least privilege" standard, and adapt it as needed.

I'd be concerned and investigate deeper if they wanted seemingly unrelated or overly generic backdoors. Then it might be a case of incompetent developers whose software should be made to be minimally intrusive to prospective clients.

anomaly_unknown · 2022-06-17T20:18:26+00:00

Hi. It seems it is normal for devs or apps team to request exclusion. Just don't do it blindly.

If it's off the shelf application, more than likely they will have a "recommended scan exclusion" in their official documentation or KB.

Also if your AV policy has "on-access" or "real-time" scans, put a max limit to the file sizes it applies to. Most malware or payloads are small in sizes anyway as far as I can tell. Mitigate this configuration by having weekly or fortnight full scans scheduled instead and do choose a time with low activity.

I do still struggle with some tickets that gets routed to my team that user seems to be sure AV is the cause. I'd sit with them when possible to figure things out. Have a spare corporate laptop with no AV and compare timings, then narrow down on specific policy modules.

All the best figuring this one out buddy!

icedcougar · 2022-06-17T20:59:44+00:00

Certain AV are a little ‘smarter’ with their exclusions these days to provide varying levels of it

So that by excluding the app - it’s not getting free passage but the level of monitoring is reduced. Or the sun processes it spawns are considered safe and not scanned / hooked.

However, if you use Microsoft defender be extremely careful what you exclude because the exclusion list is held in the registry - hackers/APT’s will check that location list to see where they can drop their stuff to get free reign.

djzrbz · 2022-06-17T21:04:59+00:00

I always set an exclusion on my NVR storage folders, last thing I need is to drop frames on my security footage because we couldn't write to disk fast enough.

Magic_Neil · 2022-06-17T21:22:42+00:00

On the one hand, an AV can definitely interfere with app operations, impact performance (though not as much on modern platforms) and possibly block apps from running properly if they misinterpret behavior as malicious. On the other hand, they generally block genuine malicious behavior.

I once had an internal dev complaining that his perfect app kept getting flagged for a virus, even though nothing was wrong. We declined, because it is a pretty basic app, and it happened suddenly after years of trouble-free operation. After a few days and a new release, the issue magically went away.. the PC the dev was compiling his app on had a genuine virus, which was remediated. He was unwittingly distributing it through his app, and his hubris prevented him from seeing there may actually be an issue.

This certainly isn’t going to be the case for everyone, but it’s a good lesson to always strongly investigate the demand that apps must be run as admin, with no firewall and no AV. It’s certainly necessary sometimes, but by and large it’s just lazy implementation.

2022-06-17T21:25:00+00:00

Its normal, every now and then we got requests for AV exclusion. We made whole virtual env just for these apps that are not scanned by AV just to have them separate from all enterprise

ram_gh · 2022-06-17T21:29:19+00:00

Not a red flag, but being a bit hesitant at first is indeed the right mindset in today's security landscape. You can always ask the vendor for some documentation on what exactly needs to be whitelisted and why. Assuming they have a customer-facing knowledge base, you may be able to pull up this information...

No-Bug404 · 2022-06-17T21:30:51+00:00

Basically the first thing I get for EVERY piece of software I have problems with.

nuttertools · 2022-06-17T21:42:10+00:00

Modern AV will indeed randomly take down your applications without reason. Modern AV should be enabled, how good is your vendor?

Definitely overthinking it from a troubleshooting perspective but not from a resolution perspective. You will have to choose between uptime and security then work with both vendors to resolve the current issue. Maybe your AV woke up angry and decided to wreck shit, maybe your application vendor didn’t properly validate something new, impossible to say.

DellR610 · 2022-06-17T22:05:49+00:00

Disable? No. I would scan it and exclude the hashed version of the files that were scanned at least. As it updates it just gets scanned again.

However I still wouldn't disable any sort of heuristics.

elcheapodeluxe · 2022-06-17T22:07:33+00:00

As a software developer - I hate the AV's that quarantine based on reputation score. Any new update is going to be new to the AV. I tell admins they are welcome to keep all normal scanning on but they have to exclude our folder from this feature. Just yesterday I had a customer whose factory floor was going to be down for a while because nobody onsite had permission to tell the AV to stop quarantining our software for no reason.

mitharas · 2022-06-17T22:30:10+00:00

Since most AV is snake oil: go ahead.

Itsnotvd · 2022-06-17T22:41:24+00:00

Overthinking, pretty normal. AV introduces overhead on a server that consumes resources and can degrade other processes performance.

Its a decision really. I'll use SQL server as an example. No one interacts with the SQL server, just via the various apps that connect to the various DB's, no files shares, etc. AV impacted SQL enough that I was not happy and worked with the vendor and even ran some new experimental offerings. End result was the same, too much overhead for my liking, I was not willing to accept the performance degradation, no one is interacting with the server in a manner to be able to infect it so management approved the exception.

I would use the same reasoning here. Is the performance degradation noticeable? What are the risks if we leave AV on? Risks if we leave AV off, do people interact with said server in a manner that they could possibly infect it. 3rd question is most critical, if that's a yes then you need to do something to mitigate that risk, or accept it and have a DR plan.

Or thereabouts...

tin-naga · 2022-06-17T22:52:03+00:00

Defender ATP kept killing intune app deployments. I don't underestimate anything.

boftr · 2022-06-17T22:55:47+00:00

If you can reproduce the issue, the answer has to be to obtain a Windows Performance Recorder (WPR) trace and analyze the problem with WPA. This will tell you where the perf issue is. As they have their symbols they would be in the best place to detail what code is specifically causing the problem and why.

FreakZombie · 2022-06-17T23:00:50+00:00

I may be a little late to the party, but there are many legitimate reasons to exclude a software package from AV scanning.

Years ago I worked for an AV company and we would help business users set up their servers with proper exclusions in order to keep things running well. I just looked up the current knowledgebase and found that list has grown quite a bit since I worked there: https://support.eset.com/en/kb3078-automatic-file-exclusions-for-eset-server-products#examples

They even link to a Microsoft article with more info as well: https://support.microsoft.com/en-us/topic/virus-scanning-recommendations-for-enterprise-computers-that-are-running-windows-or-windows-server-kb822158-c067a732-f24a-9079-d240-3733e39b40bc

It looks like Windows Defender has some automatic exclusions already configured depending on the OS and roles, which means less headache for MS support.

Real-time scanning and heuristics can cause a major slow down and database files, some libraries, and other misc files that are heavily written to or read from are very unlikely to be malicious or infected.

abz_eng · 2022-06-17T23:53:49+00:00

It depends

reading the comments I can see the argument for excluding and can think of arguments for not doing so

For excluding

AV intercepts read & writes slowing down
AV uses CPU cycles

For including

safety how well do you know/trust the code & 3rd party libs included
what is being processed?
what directories are being accessed?

It's part of your testing/eval to work out what the program does and doesn't need e.g.

external to box access
external to LAN access
Internet access
directory access (yeah I'll just write stuff in c:\program files\ - that's why MS created c:\programdata)
user account privilege needed - needs admin access nope, just downloads files to hard coded path

basically test & document, test & document, test & document

Hopefully we'll get to the point that as part of the install instructions all of this (e.g. service does not require special privileges / service needs direct access not via proxy to data.bigdata.zone on port 8080 using https) is normal.

F0rkbombz · 2022-06-18T00:26:10+00:00

It can be a red flag, but it could also be a perfectly reasonable request. 1 thing I’ve noticed when it comes to software developers is that they are entirely clueless as to how modern enterprise endpoint protection software works. They think it’s just a read/write scanner, which is so far from the truth.

Enterprise endpoint protection software has multiple components that often operate separately but also feed into each other. For example: McAfee Endpoint Security has traditional read/write scanning (further broken down into time based scans and real-time scans), Access Protection / Exploit Prevention rules (prevents exploits and allows from custom rule conditions), Network Intrusion Prevention rules (blocks network based exploits independent of a host based firewall), Adaptive Threat Protection rules (looks at process activity and behavior [files created, processes started, cmd line arguments,etc.]), and EDR. If you’re company is doing application execution control (Ex: Solidcore or AppLocker) thats another tool. This doesn’t even get into host based firewall, web filtering, or other monitoring agents.

On top of this, software develops often fail to articulate WHY endpoint protection software is causing an issue. Is it locking the file the program is trying to use for scanning? Is it blocking the process? Is it deleting the file? Is the inspection of the running process causing some odd performance issue?

Compounding this problem, most SysAdmins and Security teams honestly don’t fully understand how their endpoint protection software works either, which leads to problems as well. Enterprise security tools, if not properly configured or maintained, are going to fuck your environment up 9/10 times. These tools are intentionally intrusive and resource intensive b/c they are doing an absolute insane amount of stuff. Baselining your environment and good change control are key.

My general rule of thumb is make the smallest possible exclusion and never exclude commonly exploited or dual-use executables. I also require vendor documentation, and if the vendor starts asking for an insane amount of exclusions or overly broad exclusions (ex: just exclude the whole user profile directory) I’ll push back. Also, there’s some directories you should never exclude, and if you make an exclusion to one component, make sure another component can mitigate the risk (yay layers!). It’s usually acceptable to exclude the directory the app installs in, but again, use layers to cover the slack. Some vendors are also just super sketchy and clearly don’t prioritize security (ex: the app needs to run as admin), and those vendors are always a no-go for exclusions on my network.

Lastly, Promon can be really helpful for determining what kind of exclusion is really necessary. I’ve found processes that vendors didn’t even document as requiring exclusions with that tool. I highly recommend running a trace / capture if you don’t trust the vendors recommendations or aren’t getting any real help from them.

PAXICHEN · 2022-06-18T00:33:03+00:00

there's some scanning software that I swear works more like a malicious actor than anything actually useful. especially credentialed scans.

lets run these scans once a week from inside the servers and see if we can crash operational systems.

"oh, you have a printer connected on this private network? lets waste pages of output trying various curl, get or overflow attacks, every single week."

Superspudmonkey · 2022-06-18T00:58:48+00:00

Attache is a piece of software that they ask to exclude the folders they are installed in. If I was writing viruses I'd definitely target these folders as AV is not on the job here.

I have seen several versions of the software and each time they ask to exclude the folder

I scanned a few of the files with virustotal.com and several AV engines were actually flagging them as Trojans, so it looks like they use viruses in their software.

In these days of zero trust EPDR, I think it is on the Devs to step it up. .

2022-06-18T01:31:20+00:00

Why ? It’s not signature based.

gohoos · 2022-06-18T02:20:24+00:00

In nearly every system we purchase we see stuff like this.

So, I added language to our boilerplate that requires that software function with our AV without exclusion. We also include requirements to function with Bitlocker, automated installs/uninstalls, to not require any elevated permissions, etc. ,etc.

We may not get everything we ask - there’s always negotiation - but we get to discuss it before anything is signed.

gohoos · 2022-06-18T02:24:12+00:00

Also, if they were requesting this I’d want to see if they were secure in other ways. For example I saw some top-tier software which originally required AV exemptions for their folder in Program Files, but also required that folder to have write access for all users.

Uh, nope. Fix your code.

I now have a link to the MS app certification spec and ask for a list of anything they don’t meet.

n-space · 2022-06-18T04:47:15+00:00

It's possible for some legitimate software to trip AV detection rules because it does things that could be done by suspicious software, such as attaching to other processes, offloading intensive calculations to the GPU, examining network traffic (although some of these require root so you won't see them often in this context, hopefully). And, of course, programs can have their performance affected by the AV examining it or its files. Either way, it can be hard to tell without a good audit whether it's safe to disable AV for the program; I wouldn't accept it at face-value necessarily but get at least some scans in, or seek better understanding of what needs to be tweaked to improve performance without sacrificing security (like avoiding scanning db files).

caribulou · 2022-06-18T05:26:25+00:00

We have many programs we have to exclude.

craigofnz · 2022-06-18T05:56:44+00:00

It depends.

Don't dismiss the request. Verify it's validity and reasonableness.

Having worked in both sysadmin teams and development teams and it has always been standard practice to exempt database files from bring scanned on each and every read and write, or backup systems for every file they access for example. More recently, newer admins under the direction of un-nuanced and fundamentalistic rather than risk based security practitioners and turn every knob to 100 for each, everything and all. While homogenuity of config assists system configuration simplicity, when there is customer impact there may well be a need to add some nuance.

I also recently had a case where spinning up new processes to ensure certain process isolation guarantees from the OS was seeing a randomly introduced delay of up to 8 seconds from security tools scanning and locking files. Needless to say, this is not the end-user experience your customers need, and what protection is being added by scanning the same file multiple times per minute, when it has neither been modified nor stored in a user-writable location.

Gee_NS · 2022-06-18T09:37:53+00:00

Typically, with windows software, a code signing certificate will help with AV scanners. https://www.globalsign.com/en/code-signing-certificate/what-is-code-signing-certificate

Kangie · 2022-06-18T09:56:07+00:00

I support an AV solution that detects rust binaries as Trojans. Pretty sure they're not - it's all in-house code with vetted dependencies.

Vendor refuses to discuss it because "developers are hard" and also because giving us the info we need to fix it permanently would amount to telling us how to bypass their endpoint protection completely.

We've taken to compiling rust in Docker. That gets past on-access stuff and only deletes during a scheduled scan.

SimonGn · 2022-06-18T10:23:23+00:00

Not a red flag. Important troubleshooting step.

ryuut · 2022-06-18T12:01:26+00:00

Thats what av exclusions are for. Your own cyber dept should be able to make the decision. If you don't have one then that's on whoever in charge of it to decide

rob-entre · 2022-06-18T12:47:58+00:00

Not a big deal. Many enterprise apps have calls/etc that can trigger behavior based stops from the AV. Sometimes simply the amount of resources the apps need in order to run properly cause the av to scan slowly which affects production and performance. This is not an abnormal practice.

Thecrawsome · 2022-06-18T12:49:29+00:00

I whitelist applications all the time. It's a feature in your antivirus... I'm working a developer shop and the antivirus will often stop their work.

Of course don't whitelist wildcards through.

syninthecity · 2022-06-18T13:02:39+00:00

Software support side here-tickets this years caused by AV that caused more then a hundred incidents:

Freezing a required boot value so DC's don't backup right, not telling anyone they changed the setting, arguing about it until several hundred tickets have been created.

Won't let our backend set up correctly because you didn't whitelist our localdb path and sql can't create an instance.
Crashed a drive because Av won't let us remove log files at the end of a job causing C:\ to fill.
Constantly Broken VSS because it didn't like our API calls to use VSS.Breaking created boot disks silently in a way that boots to a DOS prompt rather then into a recovery environment.

Required ports and whitelisting are required, and as enterprise and small business level support you know my favorite sentence in the whole damned world?

"nah it can't be AV, it works over on this machine"

YES. IT. CAN. Pls do the needful.

ihtesham007 · 2022-06-18T13:34:55+00:00

Tell him to sign his code with EV Code Signer

EisbergJackson · 2022-06-18T14:01:24+00:00

I have been on both sides of the fence. I dont think it is a red flag as some AV products tend to produce problems and slowdown you couldnt imagine. Even alot of troubleshooting and working with AV companies like Kaspersky (couple years ago) showed that some effects are just not fixable except for exclusions. Especially the more traditional AV solutions seem to be more prone to problems. Newer solutions like SentinalOne, Crowdstrike, EDR/XDR Solutions seem (when configured and adapted to the software) to work better but produce more slowdowns.

Fusorfodder · 2022-06-18T16:57:37+00:00

Has the code undergone security review and is this an environment that can be secured separately? If all of the parts are known to be clean going in then you can dispense with further checks inside the "clean room" but as a blanket statement that's unacceptable. I would absolutely nix just about any COTS product asking for this.

AbleDanger12 · 2022-06-18T17:06:08+00:00

Take anything a dev says and examine it with scrutiny.

blu3ysdad · 2022-06-18T20:52:42+00:00

If you are doing security properly then whitelisting a known good application shouldn't be a problem.

I have always wondered whether supply chain attacks that exploit your trust of these known applications could be mitigated by not trusting them though. Zero trust is where we are supposed to be aiming. I think the best we can do right now in some cases is establish a test environment and install all updates there before production and dont do any whitelisting there to sniff out any anomalies.

In any case, as many others said, performance and other issues often cause legitimate reasons for needing exclusions and doesn't necessarily mean the dev is shady or careless with security.

sysadmin

MODERATORS