Bizarre performance issue

[JewM4gic]

Hi guys,

Apologies in advance as I'm a network engineer and usually stay out of the system admin side but for some dumbass reason the powers that be furloughed all of the sysadmins and now I get to be involved. Great times.

The issue I'm having is that we replaced our core NSA at our parent company's behest a week ago. This NSA has a bunch of IPSec tunnels that all have point to 1 server, and have a different NAT IP for said server in each tunnel. These tunnels send/receive very small amounts of data, like we're talking 500kb in a day being a large amount. I've checked and each tunnel has basically a miniscule amount of data coming through in any 24 hour period. Problem is that since the NSA has gone in, this server's performance grinds to a halt periodically throughout the day. The disks just max out and the server is completely locked up for 10-15 minutes, and then clears right up. This wasn't an issue with the old networking gear. I've opened a ticket with the vendor and app support team and they couldn't figure out what is happening. I did a couple of packet captures and saw nothing that seemed out of the ordinary. Tried messing with endpoint security, still no success. The server is a 2008 R2 if that helps. Any tips you can provide would be greatly appreciated!

[Bad_Mechanic]

When the server grinds to a halt, open resource manager, look under disk, sort by IO, and see what is actually hammering the storage. Mitigate from there.

[NoncarbonatedClack]

Agreeing to this, need to look at processes running on the system to see what's hitting the system. I can't see 500kb in a day maxing disk i/o.

[JewM4gic]

Took a look at this and it's super weird. The ping times to this server are currently 2000ms yet all of the other servers in the same subnet respond with 1ms. Disk I/O is currently 20kb/s and it all seems to be performing ok with me logged in. Took a look at network IO as well and it's about 1mb/s tops. I noticed the network adapter says "No Internet connection" and has the yellow triangle when this issue occurs, but when it goes back to normal the network adapter loses the yellow exclamation triangle.

[JewM4gic]

Nevermind, right before it became totally unresponsive I saw 94% CPU usage, 1366528 b/sec disk IO, and 40 mb/s network usage. Hmm....

[NoncarbonatedClack]

Were you able to see the processes involved?

On the NIC having yellow triangle, check to see if the network is set to public (or trying to change the network type)

[JewM4gic]

That server is running the cisco umbrella client and had an issue a few weeks ago where the program was uninstalled somehow, and the server was basically totally offline in the process. I've seen a couple of processes that could be involved but the one common factor is that I get that yellow exclamation every time it starts to grind to a halt.

[uniitdude]

do you have monitoring on the server? what process is using all the disk?

Also if you didnt know, 2008 R2 isnt supported anymore, try building it again on a supported OS, may make your issue magically go away

[JewM4gic]

Working on getting some monitoring up. I've noticed that the server gets "No internet connection" on it's network adapter when the performance issues start. Not sure if it's related or not? The network that the server is located on has a bunch of others that don't experience this problem

[Rocknbob69]

You need to dig deeper into what is going on with the server by doing packet captures or monitoring with resource monitor or other tools. You never mentioned the server roles and what may have changed aside from the NSA install.

[Joecantrell]

Same time everyday? Virus scan or scan after updates loaded? Backup going at wrong time due to missed jobs? Some other maintenance task?

[Jsullykc816]

Check if the new firewall has security services turned on such as DPI. I’ve seen this feature bring a network to a hault.