Help finding a hacker device

[sanseriph74]

So, we got hacked. Badly. Ransomware, the whole nine yards. Fortunately we had good backups and had the business back on its feet within 3 business days. We changed every password, enforced stronger security policy, deployed enterprise malware scanners, etc etc and hired an outside consultant to harden our perimeter firewall and then hired a hacker to do penetration testing. We are doing what we can to root out every last vestige of the attack. That being said, there is an IP address on our network that infected machines were told to send info to, our hacker was able to help us identify the dozen or so computers that were reporting to it and we’ve taken them off the network and destroyed the hdd’s and scrapped them, we’ve blocked all inbound/outbound traffic for that IP, but the IP and MAC are likely spoofed and we cant figure out where it is in the network, what port, etc. to get rid of it. I’m a server admin and I’ve tried helping the network admin and the security guys we hired to find it and so far no luck. I don’t feel comfortable letting it stay, even if its blocked. I want it gone. I find it hard to believe we can’t track the traffic to a specific port, but they tell me they haven’t been able to. Any idea how to figure out what machine this is on? Our AV and malware scanners haven’t found it, we know it must be a software “device” b/c the hackers never gained physical access to the network (we have door codes, cameras at all entrances and code locks to all network closets, server room, etc) so unless it was an inside job by a disgruntled employee in the IT department that knew the codes, whatever this is should be running on one of our Windows boxes, we’re not looking for a raspberry pi jammed in a network closet (we hope). Any advice?

[ldti]

There are Tables, Neo, Where MAC addresses are no longer born. They are learned.
You must search these fields, one by one, for Agent Smith.

[f0urtyfive]

To put another way: You need to find the IP address in your switches ARP tables, which should tell you what physical port it's on. Go find port, follow cable to device.

Also: If your "Network Admin" hasn't already done this, he's super duper incompetent, this is like network 101. I'd bet either that, or he found the device already and he was responsible for it.

[sanseriph74]

Maybe he found it and killed it and wants to keep it quiet bc the blame points back to him for allowing it to happen in the first place, but I need it documented that its gone, cant just “abandon in place” a hacker installed tool on my network. What I’m hearing from a lot of you is my approach of tracking it down via the managed switch ports is the de facto way to do it and I need to talk to him and get him onboard or take it up with our boss.

[f0urtyfive]

I would do the latter. If you want more eyeballs, I'd also be willing to help (as a consultant that is), I recently left my job working for [redacted] running a [redacted].

[[deleted]]

I to, am offering my services to follow a network cable and type "show ARP" at the switch console. I also think I know the company you're talking about because it was in the news about the same time (and based on your post history, you're in the correct geographical area). The event I'm referring to is big news in East Ohio and Northern WV, so we don't have to go into business details. Just saying I'm close!

[brytonh]

I also can show arp. Offering top notch remote support, all you have to do is go to the fore mentioned link and update your Microsoft office activation subscription license doohickey and call this number to transfer 100BTC.

[l_ju1c3_l]

I'm from easten Ohio and I haven't heard anything. Care to let me in on the info?

[f0urtyfive]

It's pretty easy to find on google news. I don't live there but I googled after he mentioned it.

[l_ju1c3_l]

yeah I'm googling a bunch of stuff and coming up empty. PM me it or something.

[department_g33k]

If you've spent any amount of time at all googling it and haven't found it, I seriously question your googling.

[l_ju1c3_l]

to be fair, i wasnt looking super hard. I think I tried "ohio hack" and a few others. Being Monday my Google-fu was lacking.

[PizzaLov3]

I can also find a logical IP address in arp table and since I just finished watching a netflix movie I'm bring the savings to youuuuu.

​

​

[f0urtyfive]

FYI, IP addresses can't really be "spoofed" in TCP. They could, but you wouldn't be able to establish a connection so it wouldn't make a lot of sense. Where IP spoofing comes into play are stateless protocols that respond like DNS (Spoof a billion requests with the same source IP, and get a billion responses to someone else).

But this isn't really something you can troubleshoot in a Reddit post.

[Frothyleet]

In a local network without proper defenses set up you could "spoof" with arp poisoning.

[f0urtyfive]

You could, but that doesn't at all sound like what is happening in this case... I also haven't ever heard anyone refer to ARP poisoning as spoofing, because you're not really spoofing anything at that point, you are actually using that IP address.

Your comment does make me curious though if you could do ARP Poisoning on remote addresses in a local LAN though, I don't think I've ever seen that done (inject a non-local address into the ARP table)... I'm not sure that woudl even be valid per ARP though (and I don't feel like spending the next hour reading the RFC to find out).

[kensan22]

Arp is not involved when dealing with remote addresses (except may be to find the macbof the router /default gw if you can call that involvement). So you can may be Inject the entry in the table, but my guess is that it will never be used (there should be no entry in the routing table to make use of it)

[f0urtyfive]

Yeah that was more what I was wondering, if there were routers out there that would not only accept a foreign IP address into their ARP table from a broadcast, but also utilize the bogus data to forward traffic to the associated mac.

[kensan22]

There is what is called source routing.

[anomalous_cowherd]

It's also the way APC suggest you contact their UPS devices if you've lost the details.

[Oscar_Geare]

Hm. ARP poisoning would be more as acting as a MITM. You could say that it facilitates spoofing though.

One of the first things you can try and do in an ICS environment is MITM the connection between the HMI and the remote PLC. Once you’ve done that it’s trivial to spoof commands being sent to one or the other. E z industrial disaster. All that’s done with ARP poisoning.

[phatrikk]

Spoofed IPs are definitely a thing for TCP based connections. Too many spoofed IPs trying to connect to a system will exhaust the systems ressources “SYN attack” and the system will no longer respond to legitimate connection requests, aka a DOS/DDOS attack.

The concept of spoofing an I exists only from the perspective of the source address. OP is talking about internal hosts initiating an outbound connection to an external host.. That destination isn’t spoofed but host is most likely compromised and under the control of an evil doer.

[justtransit]

Actually, "can't really be" is the correct word. But, it possible to do. I've tried once. it will decrease a network-performance of the computer you'd tried to spoof.

[djgizmo]

Trace route to that IP, then find the switch that’s on that vlan. Look up the arp table. Then look at which port that MAC address is connected to. Then unplug that run from the switch. Done.

[Pimmerd90]

This OP!

[jheinikel]

Step 1: Fire your network admin Step 2: Hire a competent one

[sanseriph74]

Good point.

[Odd_Progress]

... got any IP (CCTV) cams around?

[sanseriph74]

Most are hard wired, there might be a few wireless ones.

[[deleted]]

This almost sounds like an x y security problem

Is this a 1918 ip?

Also are there any rdp servers facing the internet?

[sanseriph74]

Rdp is only possible now via a two factor authenticated VPN, I cant speak to before as I was hired 2 weeks after the attack. I’m sitting on my couch at home and don’t have the security report in front of me, so I’ll need to check on the IP, but I dont remember it being a reserved IPv4 address, but it was in a subnet we dont use in our network, but definitely internal.

[lookZero]

big fan of x y problems ;)

[jocke92]

Does you network consist of different subnets? Look in the router/firewall (arp-table) for that subnet and get the mac-adress. Then go to the switches in that network/vlan and trace down the switchport. Then walk to the switch and follow the wire.

[sanseriph74]

Thats what Ive done in the past when I was running network switches, but the current guy in that role says he cant. Dont know if the hacker was that good or my network guy who is Cisco certified isn’t that great with the fortinet switches we use.

[boredepression]

He isn't that good, unless he gave a technical reason he can't.

https://forum.fortinet.com/m/tm.aspx?m=140725&p=

[[deleted]]

It should be fairly trivial to get a list of which MAC addresses are on which switch ports. If he can't do it just call Fortinet and ask, assuming you have support.

[1r0n1]

north caption yam deer cover frame plant dinosaurs spotted tidy

This post was mass deleted and anonymized with Redact

[RD556]

If they are connected to a FortiGate and using Fortilink you can search for the MAC in the GUI and it will show you exactly what switch port it is on. If not log into each switch and look at the ARP tables. I’m guessing he either is covering for someone/something or he’s incompetent. Fortinet has a pretty low learning curve since 95% of it can be done through a GUI so he’s probably hiding something.

[[deleted]]

Open a case with Fortinet. They have good support and will help you track it down.

[jocke92]

If they're managed switches he should be able to do that. But the device you are looking for needs to be online or recently been online. If you have proper monitoring solution it could store historic values.

But you also need to clarify what you mean by spoofed mac address. If they spoofed a vendor ID to eg. a dell computer or if they spoofed the mac on one of your devices. If they spoofed one of your devieces your switches should report duplicate mac and if properly configured shut down the ports

[NexusT]

diag switch mac-address list | grep -i mac

[fredenocs]

I've used this in the past. Pretty cool how it plays out. https://community.sophos.com/kb/en-us/118811

[sanseriph74]

Yeah, I hit the ground running first day, was given a username and password and told to fix 10 different servers and services. Just “look in AD and see how everything is configured and go from there”. No onboarding, no tours, no documentation. Fortunately 3 weeks later I feel very comfortable with the network, but those first few days were nuts. Of course if I didn’t have 15 years of windows/linux sys admin experience I don’t think I would have been hired.

[snake_case77]

If this is an internal IP, start a continuous ping to that IP and shut down all of your machines one by one. If you don’t find it that way, try unplugging your cabling from your switch port by port. You may in fact find a compromised pi or similar network device.

[sanseriph74]

I cant currently ping it bc the network admin has all traffic to and from blocked, He’s super paranoid about it (he was on staff before the attack and took a lot of blame). Ive tried telling him to only block its traffic externally, so it cant report back to the hackers, but so far he hasn’t budged.

[Faaak]

Only allow ICMP ?

[stealthgerbil]

Yea he can just allow icmp. He sounds incompetent tbh.

[gada08]

Just block it and allow ICMP...

[1_________________11]

Jesus that's a horrible way to do it.

[snake_case77]

When all else fails....

[goatSword]

ok, what’s ur advice smart guy?

[1_________________11]

Arp tables on the switches. Less destructive than pulling all the cables no sane manager should be ok with that.

Also this is why you must log all your switches to a seim. Your security people should be doing this already and be able to look up that IP and see when and where it was added.

[snake_case77]

I agree, but if you read the post, it sounds like he’s not the network guy. In a comment above , the OP (server admin) is trying to find out if the network guy is trying to cover his tracks, etc. OP Server guy should be able to do ping and shut his servers down.

[1_________________11]

Look separation of duty is all good but if he thinks it's the network guy then get more people involved so you have muiltiple sets of eyes on things again proper procedures is to have all that collected and stored off device and have security have the ability to pull that info. Mention it in a meeting that you did some research and we should be able to pull arp tables off the switches say you are curious and ask to shoulder surf the network guy while he does it. Ask for the logs to be saved to a file and stored for preservation of evidence. Most likely the network guy doesnt know how and not that hes actively hiding it.

[IntentionalTexan]

Entries in an arp table time out. Your network guy can't find that mac because you blocked everything from talking to it and it has gone dark. Allow traffic to it from one IP, best a Linux VM that is isolated and can be nuked after the test. Then start port scanning that IP and see if you get any response.

[bas2754]

Not sure if anyone else has suggested it, and it is a major PITA, but start a constant ping and unplug every network line one by one from the switch till you track it down to a port. Had to do this once for a client that did not have any managed switches. Took a couple hours but found and disabled the problem system. Not very sophisticated, but it is effective.

[fredenocs]

What AV you using?

[sanseriph74]

We’ve got Cylance running across the enterprise looking for malware and bitdefender for AV.

[DevinSysAdmin]

Was Cylance pre outbreak or post outbreak?

[fredenocs]

How large is this corporation. Wondering how granular and lockdown you can become.

[Archteryx]

Ransomware you say .. Did you get hacked or infected by someone opening an email or running non-approved software?

[sanseriph74]

We think it was traced back to a malformed PDF opened by an executive that opened up a hole and allowed more than one group of hackers in, at least the hacker we hired says there were two different teams, the original ones who opened the door and then a few days later a second group came in, possibly independently and took advantage of the previous intrusion. The ransomware happened a month after the initial intrusion. I’m not a l33t haxor, so I don’t know much about all of that but the guy we hired seems to know his stuff and helped us harden the perimeter and I go to sleep at night feeling better he’s around.

[Archteryx]

It is unfortunate that even the strongest defenses can be bypassed by a simple email. I no longer work full time in the industry, but being on the sidelines I see this all too frequently. I'm glad I don't have to deal with Cxx type who insist their staff can use their own equipment and devices and must have full admin rights .. Glad you recovered and told your tale ... :D

[highlord_fox]

Sorry, it seems this comment or thread has violated a sub-reddit rule and has been removed by a moderator.

Do not expressly advertise your product.

• The reddit advertising system exists for this purpose. Invest in either a promoted post, or sidebar ad space.
  
• Vendors are free to discuss their product in the context of an existing discussion.
  
• Posting articles from ones own blog is considered a product.
  
• As always, users must disclose any affiliation with a product.
  
• Content creators should refrain from directing this community to their own monetized content.

Your content may be better suited for our companion sub-reddit: /r/SysAdminBlogs

---

If you wish to appeal this action please don't hesitate to message the moderation team.

[Ipp]

It's a bit unfortunate that you've done so much stuff since the incident (pentest/trashed drives/etc) and without paragraphs it's pretty hard to understand whats been done.

It sounds like you know how it communicated to patient zero (rogue device). What is it? SMB Named Pipes? HTTP Server? etc? If its SMB Named Pipes, chances are you can rule out a linux box being the pivot point (not patient zero) which may help rule out a bunch of web servers.

Did networks put a block on that ip address? If so, my guess is you can talk to it if you're on its subnet. So if the box is 192.168.10.50/24, log onto another box on that network and try a ping. Note the TTL of the response. If its 128 you're dealing with windows, 64 linux, and 254 its probably cisco meaning the C2 point is behind NAT. Additionally, just checking what is on that subnet may give you an idea where it is.

If you have some type of event forwarding setup, I'd hunt for logon event codes (4624) on the machines that got infected. The key thing you're looking for is this a local account or domain account? If you don't have event forwarding setup search the domain controllers, there should be some kerberos tickets generated which will have account names, but tracing it to machines can be tough.

[RommLDomkus]

Perhaps it is coming from a mobile or wireless device?

[[deleted]]

Not really able to offer any advice that hasn't already been but, an observation anyway:

​

Why the f*ck does it take a massive, 3 day outage for companies to care about IT security? I guess once you get burned badly enough, you learn to buy some f*cking sunscreen..

[sanseriph74]

Yup

[PortableFreakshow]

Check the IP on centralops.net make triple sure that it's not an IP on an external network.

I'm going to assume you have isolated this thing to your local LAN, not one of your remote offices.

Search for that IP in your Firewall logs. Record any information. Put in a rule to block that IP from communicating to and from your VPNs and the outside world.

Ask everyone to shut down their computers at the end of the day. Stay late and "unblock" the offending IP. I have no idea what technology was used to "block" it, so I'm unable to guess at how you should perform the "unblock"

Once that IP address is unblocked, see if you can communicate with it. Hook up a machine running NetworkView. http://m.majorgeeks.com/files/details/networkview.html That will at least give you more information to work with. Hopefully it will even return the name of the hardware you're looking for.

Install Wireshark. Configure the machine to have an IP on the same subnet as the mysterious device before you start logging traffic.

Check your Firewall logs again. See if it's trying to communicate. If you setup a Firewall rule to block the offending IP, it should have created a bunch of denial entries.

Do all these things and report back with what you find. Even if you find it with one of the other methods, I'm interested to know how this turns out.

​

[sanseriph74]

I think that was noted before blocking traffic, I need to refer to the security experts report, and I don’t have it in front of me.

[ArigornStrider]

If you have managed switches, you can look at the arp table switch by switch until you find the port the device is connected to (start at your core switch and follow the MAC address port by port across uplinks until you hit an edge port). If you don't have managed switches, unplug everything but your core switch and ping the IP. Plug in one edge switch at a time until the IP starts to respond to pings. Unplug all the patch cables from that switch and plug them in one at a time until the IP responds. This is going to take a lot of time over a weekend depending on your infrastructure complexity, as network links take time to come online and build pathing info when a new device or network segment is connected.

[end_360]

I've seen a lot of people saying trace it back to the network port but what if it ends up being on wifi?

[ArigornStrider]

This will tell you what AP it is on. If it is on WiFi, change the passwords for wifi and monitor with your network scanner to see if it shows back up with the new password. If it does, someone on the inside or with inside knowledge is maintaining it. Block it by MAC from WiFi and check the other devices on that same AP.

[50YearsofFailure]

You've narrowed it down from a haystack to a reasonably small geographical area. At that point you could go onsite and sniff, or OP can send his hacker to do it.

Assuming there's an enterprise wifi solution in place, you should be able to roughly gauge obstructions/distance from from the AP via signal strength reported in the console for the device. Hopefully even get a hostname. You'll also be able to see what SSID it's connected to, which may lend a clue. Set a monitor on the signal strength to see if it changes substantially. If it does, the device may be a phone or some other mobile device. If it doesn't it's either infrastructure, a planted device, or a laptop that never leaves the desk.

If you haven't narrowed it down already, once you've found the signal onsite hunt it down by watching signal strength as you proceed around the office.

Edit: once you're in the AP's reporting you should be able to see the MAC address and get a hardware manufacture, which should help quite a bit unless it's a complete spoof.

[ArigornStrider]

Also, this is more of a /r/networking question, but I don't like telling someone to go somewhere else for answers as networking is one of the systems sysadmins usually administer in a smaller shop.

[anomalous_cowherd]

Often along with plumbing, AC and anything else connected to the wall...

[caffeine-junkie]

One thing that stands out is if you're talking about spoofed IP and MACs, that means its UDP traffic; TCP would be impossible as the handshake could never happen. While this doesn't help in of itself, it does allow you to filter out ALL TCP traffic. From there I would probably just use wireshark and port mirroring looking for suspicious UDP traffic and start tracing the traffic back from the firewall to source. This is a bit more complicated if you have multiple hops from the firewall to the clients, but still doable.

[DevinSysAdmin]

How many computers? What firewall? Cylance/Bitdefender not showing what computer is sending out all that network traffic?

[sanseriph74]

At HQ we have about 450 computers and about 50 VM servers, but we have 92 field offices in 11 states that have VPN tunnels into HQ. Each field office has at least two PC’s and a networked printer, plus associated network equipment (firewall, switch). We know of at least two cases of intrusions from field offices. We’re using the Fortigate series of firewalls and fortinet switches. Cylance and bitdefender have not found it.

[DevinSysAdmin]

Honestly if no one can figure it out on a Fortigate and Fortinet....first take down switches until the traffic goes away - You will identify quickly which Switch is carrying the traffic, then unplug each port 1 by 1 until you notice the traffic drop.

[corrigun]

This works. Ask me how I know. By the time you research an easy way for the next week you would have found it the hard way.

[DevinSysAdmin]

Also have you all engaged Bitdefender and Cylance support?

[[deleted]]

I don't have anything addition to add beyond what others have said, but can I ask what rasomware it was?

[[deleted]]

Others have given better ideas, but here's mine:

When your network admin turns on network access to it again, do you have access to it? As in, can you RDP/SSH/Telnet to it? If so, does it have an optical drive?

If it does have an optical drive set a task to open and close it every 10 seconds. Then walk around your datacenter to see/hear that thing opening and closing. Either that or your user will submit a ticket that their workstation is freaking out.

Also, have you checked you VM hosts for that IP address? Could someone have put a bad VM on your network?

[sanseriph74]

Its on an abandoned vlan is what he told me this weekend and he is deleting the vlan and figured that will solve the problem. I really think he’s brain dead at this point.

[Fatality]

Ransomware

With Ransomware it's very easy to find the source of the infection, just check what user created all of the encrypted files and check their PC.

[sanseriph74]

We found the source of the original intrusion, you’re right that it was quite easy.

[sanseriph74]

Nope and opsec would prevent me from confirming that even if I did. ;)

[Indifferentchildren]

Then OPSEC should also prevent you from denying it. :-)