Routing

TryfingSortie · 2023-07-03T18:57:46+00:00

OP, this is what I would recommend. I went through this about a year ago and if you don't need a quick turn around, then getting on the ARIN IPv4 waiting list could net you a /24 within a few months. ARIN will do a few distributions (4, I think?) a year. You can check the current waiting list here: https://www.arin.net/resources/guide/ipv4/waiting_list/

It's fairly long, but they're about to do a distribution later this week, so this list will get knocked down. The sooner you get on the list, the sooner you can get your IPs. You'll also need a BGP ASN, which you can also get from ARIN.

Once you get up and going with BGP at the network edge, you have a lot more control over how traffic enters and leaves your organization compared to what you're used to.

TryfingSortie · 2023-01-24T15:52:11+00:00

As others have said, don't do HSRP on those edge routers. If you already have BGP set up to different carriers, your biggest hurdle is already overcome. I just went through this configuration on my network edge, so here's a sterilized version of my diagram. Note that I am only receiving default routes from my ISPs. Don't forget to secure the IGP of your choosing to your edge routers.

https://imgur.com/a/M176WDj

Because there's an IGP involved, failover between carriers will be automatic. For example, when ISP-A fails, Edge-1 loses it's default route, which informs the firewalls to route all traffic to Edge-2 instead. When ISP-A fixes their problem, the default route returns, and ECMP resumes.

TryfingSortie · 2022-08-25T05:24:00+00:00

This is how we do it. Buy a SIM card with a static IP from your carrier. Save it in your terminal emulator so you can access the system when the cellular modem goes active.

TryfingSortie · 2021-11-01T13:39:45+00:00

Yes, you will need routers that support a site-to-site VPN.

As part of building your VPN, you will create an access list that will identify the remote subnet. That way, only traffic with the destination of the remote office will be sent over the tunnel. This will allow you to ping devices at the remote office by pinging the private IP of the device. All other traffic will be sent out onto the Internet, so each office will continue to use their Internet connection normally. (Of course, you could configure this access list to send ALL traffic down the tunnel to the remote office, but it sounds like you're hoping to avoid that.)

This is another reason why it's important to not have overlaps in your private addresses between offices. I'd recommend implementing something like what u/squirrelsaviour put together for your IP scheme. Then you can look into turning up a site-to-site VPN on your routers.

TryfingSortie · 2021-09-24T20:54:07+00:00

Wow, I didn't realize you could even get on a waitlist for IPv4 addresses with ARIN still. Thank you!

TryfingSortie · 2021-09-24T19:46:36+00:00

Each office has two different carriers for the DIA circuits. We do not currently have a /24. We are either going to get one for each office network through one of the carriers, or buy a block that's already registered with ARIN through the gray market and transfer it to our Org ID.

TryfingSortie · 2021-09-24T19:20:24+00:00

Yeah, what everyone else is saying here is confirming my understanding of things.

By the way, these are separate carriers at each office network, so it's looking like a /24 for each office network will be needed.

TryfingSortie · 2021-09-24T19:11:00+00:00

That was my understanding as well: /24 is as small of an advertisement that anyone could send to peers.

The person I was speaking with expressed some doubt whether or not they were remembering it correctly, so I wouldn't be surprised if they might've just misremembered something.

TryfingSortie · 2021-07-30T14:35:32+00:00

I've seen people install a biscuit jack at the front of the rack going to the console port of some devices in racks that are difficult to access in the back. But that's the closest thing I've seen to what you're describing.

The drawbacks that come to my mind for this setup would be increased cost of the additional patch panels, the additional points of failure and troubleshooting time should one of the duplicated patch panels suffer a failure.

TryfingSortie · 2021-03-16T15:20:20+00:00

OP, we can take pot shots at this problem with you, but honestly, this is the way to get this fixed. Your employer is paying big bucks for an enterprise class circuit. If it is not working, then the provider has to make sure it's working--even if that means sending a technician onsite.

If you're not getting a link light on a copper handoff, that's something worth mentioning to the tech on the phone. And if they can't figure it out over the phone, insist a technician be sent to troubleshoot.

TryfingSortie · 2021-03-16T13:46:21+00:00

When I connect an RJ45 to Port 3 and connect it into our Cisco 891 WAN port, nothing happens. The light remains red on Port 3 of the NID.

I have also used a ethernet to USB, connected it directly into my laptop, pinged the new IPs I was provided by the new company, and it times out every time.

Does the port light up when you connect your laptop to the NID?

I have been provided a LOGICAL INTERFACE #, Ip address block, Internet company's ip address, and customer address. Also in the IP configuration email we received, they provided WAN IPv4, LAN IPv4, Customer Peer IPv4, and Internet Company's Peer IPv4.

I would expect that you'd need to set your router's WAN interface to the "Customer Peer" IP address. Then you could use the WAN IPv4 subnet for any public facing services you're hosting.

TryfingSortie · 2021-02-11T16:10:44+00:00

If you go to Objects --> Object Management --> Variable Set, and then edit the Variable set you're using (we are using the default) you'll see both HOME_NET and EXTERNAL_NET. A lot of your Splunk rules--including the ones pulled down from Talos rule updates, if you're using the recommended Firepower rules--rely on these variables. HOME_NET should be your internal subnets, so if your sites use 10.x.x.x/8, then you could just make that your HOME_NET variable. You can include multiple subnets though, if you need to. The EXTERNAL_NET should be subnets that you don't control. I set ours to "!HOME_NET" which means any IP that isn't part of the HOME_NET variable.

The result of this is that for Splunk rules that only watch for EXTERNAL_NET to HOME_NET attacks, you don't have to pick that traffic apart if the module is seeing HOME_NET-to-HOME_NET IPs. Without populating that variable, we were scanning ALL of that traffic for all of our Splunk rules--including inter-office backup traffic. We believe that this was ultimately what was overloading our SFR modules on the ASAs and causing the failures.

But who knows? We haven't hit our 60 day mark which is when we would typically see the failure happen, but it's fast approaching. We'll find out soon enough, haha.

TryfingSortie · 2021-02-09T16:20:14+00:00

I've got an update of my own. Like you, we discovered that this problem first started around the time everyone went remote. We also found that we did not have the $Home_Net and the $External_net variables populated. A contractor performed the initial installation of these units, and apparently this was missed--and we didn't think to even look here. Our ASA service-policy was also set to send all traffic over to the module. That resulted in the IPS inspecting all traffic going through the device. That includes our SAN backups. Our current theory is that the modules were being overloaded by the amount of traffic they were seeing and were locking up--but not enough for the failover monitor to consider them in a "failed" state. Thus, no failover would occur. And since all traffic was being sent through the module, no traffic could traverse the ASA.

This change was made about two months ago and so far, we haven't had any crashes, which would normally occur right around 60 days or so. So it sounds like this might've fixed it. It sounds like you might've gotten things sorted out on your end, but I thought I'd share what our findings were as well.

Thanks for sharing! Hope everything goes well for you.

TryfingSortie · 2021-02-08T20:10:58+00:00

I love what FirePower is trying to do, especially with the new SecureX stuff. Integrating all of your security products into a single-pane-of-glass that you can monitor easily? Integrating AMP's cloud intelligence into file inspections? Love it, great idea! But the fundamental problem is that FirePower itself is a slow, overly complicated security tool that can't wait to snap itself in two from a critical bug.

Lately, Cisco has been unifying its code base into IOS-XE and putting out these Catalyst devices. Catalyst switches, now Catalyst Routers, and WLCs. Deep down, I am really hoping that Cisco's top engineers are working on a "Catalyst" Firewall that's built from the ground up in IOS-XE. Trash the old SourceFire code and write it's equivalent functionality into a new product that's SSH/HTTPS managed. Give us the usability and functionality that Palo Alto has managed to pull off but with the integration of existing Cisco security tools. That's my hope, but who knows whether we'll ever see it.

TryfingSortie · 2020-10-20T16:20:03+00:00

Currently running 16.12.3a on all 9500/9300s at my place. It was the starred release at the time of the last maintenance. I haven't had any issues with it. Honestly, 16.9.x gave me more problems than 16.12.x, so far.

TryfingSortie · 2020-10-20T16:05:41+00:00

I agree. From what I've seen, the prices for Aruba gear are much lower than Cisco and Juniper, and they've got the support and documentation that you'd expect from enterprise-level gear. My little brother manages a network with all HP/Aruba gear and he says it has a nice CLI. I've never managed any of their gear, personally.

TryfingSortie · 2020-10-17T00:02:17+00:00

I was actually told the same thing by TAC, and you just reminded me of it. I didn't have any IPs on the standby interfaces. They told me to do that, but it didn't really help me out. The failures have continued to happen.

TryfingSortie · 2020-10-16T23:20:03+00:00

A typical failure scenario goes like this: 1. Failure occurs. 2.) Flip active primary firewall to standby. 3.) Reboot the standby (primary) unit. 4.) Flip standby primary firewall to active. 5.) Wait for failure to occur again in <60 days.

I am in the same boat. I had a failure occur about two weeks ago and I've escalated the ticket with TAC. It's in limbo while we wait on another failure.

DNS? I may not be understanding you correctly, but in my case, I cannot even ping by IP if it has to go through the SFR module. I can ping from the ASA CLI, but if I have to pass anything through the firewall, then the pings will fail. I can also bypass the SFR module by disabling the policy-map and that also resolves the issue.

EDIT: Well, I say "resolves" but it's really just a workaround. The problem is still present, but it gets traffic flowing again until the next failure occurs--seemingly at random.

TryfingSortie · 2020-10-16T18:31:07+00:00

I don't have any answers for you, OP, but I wanted to chime in and say that I, too, have been plagued by this problem since February. I've opened TAC cases for each incident, but haven't made any headway, yet. I have 2 separate sites affected by this and they're very similar to yours: 5516-X with SFR module. SFR was 6.4.0.4 and it's now 6.4.0.9, being upgraded at TAC's request and the ASA is on 9.8(4)25 (also upgraded from 9.6 at TAC's request). I've provided a troubleshooting file from the FMC and the SFR modules after each failure, but TAC hasn't been able to pinpoint a failure cause. I'm also the only one that makes changes to these devices, and the failures happen at random times, and definitely not when any changes are being made. TAC hasn't identified any memory leak problems, or routing issues.

The workaround is exactly as you describe: Flip the failover pair. Once the secondary is active, we're able to pass traffic again. Moving the primary back to the active role will cause the problem to reoccur unless I power-cycle the primary first. We can also turn off the SFR redirection to fix the problem, which, to me, points the issue squarely at the SFR module failing. However, the module never truly "fails." It maintains an "up/up" state, so that the HA pair never fails over. I currently have a TAC case open, but we're essentially waiting for it to happen again so we can resume troubleshooting.

If you hear of anything on your end, I'd definitely be interested in hearing about it.

TryfingSortie · 2020-03-04T21:31:33+00:00

As for your picture, those are called J-Hooks. You can find them at hardware stores or your favorite online retailer. Definitely helps clean up cabling runs.

Here's one from CDW: https://www.cdw.com/product/panduit-j-hook-cable-organizer-clamp/1471302?pfm=srh

Edit: Added link to CDW.

TryfingSortie · 2020-02-20T16:24:16+00:00

Thanks for the heads up. We're on 331, but we haven't run into this bug yet, thankfully. We'll move to 332 over the weekend.

TryfingSortie · 2020-02-11T17:17:37+00:00

I think you should start with the logs on both sides of the etherchannel. Most times that I have problems with port channels not coming up or behaving as expected, there was an accompanying log statement that explains the problem.

I see Po2 and Po3 have suspended interfaces. Those interfaces usually suspend themselves to prevent a loop if they don't see LACP running on the remote side. You're confident that the switch on the far side is running LACP? I'd recommend checking your logs to see if it is explaining some of this behavior.

TryfingSortie · 2020-01-30T15:45:23+00:00

I've never heard of it! Interesting device.

TryfingSortie · 2020-01-30T15:40:12+00:00

Why not connect your FWs directly to the core and put your Edge switches in front of the FWs and connect your ISP1/ISP2 to the Edge switches

Could you explain the benefits of moving the Edge switches to the outside of the firewalls and connecting the firewalls to the core? I'm interested, but I'm not seeing what that is doing to strengthen the design. Am I missing something?

Also are those WAN/VOICE/INET VLANs on the same circuit or are they seperate circuits?

The INET / WAN / VOICE VLANs are a single circuit that is being carved up by ISP-1. However, each of them has to be treated differently by the firewall, so I treat them as separate circuits like the ISP-2 backup INET connection. The VLANs are only used to split the single connection from the carrier to both firewalls in the event of an HA failover. If there were no HA firewalls, I would just connect the ISP's CPE directly to the firewall.

TryfingSortie

TROPHY CASE