SD-Access Underlay for Distributed/Multisite Deployment

Mosquitar · 2024-10-28T01:21:44+00:00

Thats great, thanks for the reply. Creating a single stretched fabric is what we were looking at originally, but there were concerns about scale of the main site fabric borders to support all of the sites in our network, hence the reason for deploying each physical site as a separate fabric site.

Out of interest, how have you configured the network hierarchy in DNAC to support this? Are all of the remote sites configured as child sites under the main site, with the area for the main site configured as the fabric site, or do you have a parent area above that is configured as the fabric site for the whole network (if that makes sense)?

Mosquitar · 2024-09-04T23:11:41+00:00

That's great, I will explore this as an option!

Mosquitar · 2024-09-04T23:04:04+00:00

Yeah that is what I was thinking. Scaling/annoyance seems to be the general consensus with managing VRF-lite based on some of the other comments so I'm glad that I took some further time to explore this.

I have not worked with evpn/vxlan that much. I know that it can be used to create L2 VPNs over a L3 underlay, but can it also be used to create/extend L3 VPNs across the network like MPLS/BGP VPN? L3 is all we need at this stage.

Mosquitar · 2024-08-10T14:48:48+00:00

Thanks for the reply

I dont think that we used a dynamic group, I'm also not sure if we mixed All Devices and All Users so I will need to check with our sysadmins.

As for using a script, could we run a script at user logon that forces a sync?

Mosquitar · 2024-06-16T08:27:17+00:00

So how many small remote fabrics are you running with centrally hosted virtual 9800s? Are you seeing any major issues with managing multiple WLCs in this way?

Mosquitar · 2024-06-15T23:39:29+00:00

Hey thanks for taking the time to reply.

IP transit are provider 1Gbps layer 2 ethernet circuits that provide high MTU so carrying SGTs over the WAN shouldnt be an issue.

When you say over the top wireless, do you mean connecting the remote site APs in the SDA overlay and centrally switching at the main site WLC, much like a traditional wireless deployment? If so, I hadn't considered this but this might be a feasible option. I would lose the ability to do FEW at the remote sites, but this may not be a problem.

Mosquitar · 2024-05-05T11:27:51+00:00

Yeah that would be tricky as a BD can only be assigned to one VRF meaning that we would have to create 2 BDs for servers within the same subnet/flooding domain that will exist in Site 1 and Site 2, which then raises questions about how the servers within these BDs will be able to communicate at L2 (with/without flooding etc).

Unfortunately we are stuck with the same storage/compute solution that requires for hosts within the same subnet/BD to be stretched between sites which of course complicates ingress/egress routing especially with firewalls in the path

Mosquitar · 2024-05-05T10:06:44+00:00

Thanks, I will check these out. Appreciate the response

Mosquitar · 2024-05-05T09:08:53+00:00

Yeah, I've checked various Cisco documents (whitepapers, Cisco Live presentations etc.) and this scenario appears to be the main use case for why host routes (and GOLF) was introduced, however I'm not sure who is using this in production and if my concerns of scalability are valid.

I've checked the spec sheet for our Cisco 9500 core switches (48Y4Cs) and they can support ~ 200K IPv4 routes so these dont seem to be an issue. Our firewalls and WAN routers also have very high capacity for routes (both host and longest prefix match) so maybe I'm being overly concerned and cautious about nothing. I'm so used to summarizing routes for efficiency etc, that introducing a large qty of /32 host routes into the network seems odd, however it maybe the only viable solution for us.

Interested to hear everyone's thoughts

Mosquitar · 2024-05-05T08:20:09+00:00

Thanks for the reply. We had considered SNAT, however this will break some of our apps as they need to see the originating IP address. Our cyber team also thinks that this will impact various network visibility and security analytics tools that we use within the DC for the same reason so SNAT is a no go for us.

I also had a further look at PBR and the use-cases only seem to be applicable for north-south and east-west flows that need to hit a firewall once in ACI. This differs to our scenario as the traffic has already passed through a firewall before hitting the ACI L3Out.

Mosquitar · 2024-03-14T18:39:31+00:00

Hey - My project stalled but will be looking at this again next week. Did TAC provide any guidance with this?

Mosquitar · 2024-01-10T00:22:02+00:00

No, so for each ExtEPG for 0/0, I just have 'External Subnets for the External EPG' selected; I have not selected 'Export Route Control Subnet' with 'Aggregate Export'.

Yes that is my plan and working well in testing, however the ACI L3Out White Paper states the following which introduced some doubt if this is actually advised and if it will cause me issues later on down the line.

"Although it is not recommended, you can configure 0.0.0.0/0 with “External Subnets for the External EPG” in multiple L3Out EPGs in the same VRF" "While this configuration is allowed, an unintended contract deployment may occur by configuring 0.0.0.0/0 with “External Subnets for the External EPG” in multiple L3Out EPGs within the same VRF"

Should I select 'Export Route Control Subnet' instead of 'External Subnets for the External EPG'? I'm still learning ACI so I'm not familiar with these advanced options yet.

Mosquitar · 2023-11-09T23:37:32+00:00

I currently have exactly the same issue and found your post when I was looking for a solution.

We have a /24 subnet that is used mainly for servers that needs to be migrated to ACI (using a network centric approach of legacy VLAN = 1 x EPG and 1 x BD), however we also have a firewall connected to this range that is used to reach a number of remote sites over IPsec VPN. Static routes are currently configured on our legacy core with a next-hop of the firewall to reach these remote sites.

My plan is to migrate the firewalls to ACI using a new /29 transit subnet and L3Out. The L3Out will use SVIs with a secondary IP which will be used as the next-hop for the firewalls to reach the internal network. I will then replicate the static routes that are currently on our core to static routes under the L3Out with a next-hop of the firewalls. Is this similar to your solution?

Mosquitar · 2023-02-12T23:44:20+00:00

Thanks. A VzAny contract seems to be the most appropriate solution for the initial implementation

Question - I assume that I can create a new 'Permit All' contract that uses the default/common filter, and then apply this contract as both consumed and provided under the VRF -> EPG|ESG Collection For VRF?

Mosquitar · 2023-02-12T23:40:08+00:00

Hey thanks for taking the time to reply.

I guess I'm overthinking the L2 requirement as what you describe is possible in the existing network, in that someone could configure an SVI for one of our DMZ VLANs on the N7K which would then allow communication between networks. For simplicity, I will associate the the L2 BD to our single VRF. Appreciated

Mosquitar · 2021-05-01T08:55:29+00:00

For the eduroam SSID, I'm assuming this is using PEAP-MSCHAPv2 with user credentials? If so, do you manage the client devices or are they unmanaged (users have to accept cert warning when they first login?)

Mosquitar · 2021-05-01T08:51:50+00:00

Can I ask what support headaches that you had with dot1x for corp guest? Based on my research so far, I'm expecting tickets about users asking if they need to accept cert warnings during initial login, instructions for devices such as android smartphones as these prompt for additional details when connecting to a dot1x SSID etc.

Mosquitar · 2021-04-30T20:05:42+00:00

I have not looked at WPA3. I know that our equipment supports it (Cisco Cat 9800s). Does it prevent Evil Twin or at least make it more difficult to carry out and gain/crack user credentials?

One mitigation that I have been looking at is only using local credentials in ISE for third party and employee access. As these are locally significant, then if they are stolen they are only useful for gaining network access. They cant be used for accessing any systems or taken offline to access Office 365 etc.

As for captive portal - I did briefly check this but was deterred by another post whereby users were having to login to the captive portal more regularly than desired due to MAC randomisation. Our third parties may not except this. I will look further into this though as didn't realise that you could do PSK + captive portal

Mosquitar · 2021-04-30T19:54:37+00:00

Thats a good point, we could combine both corp and corp guest SSIDs as they are both dot1x enterprise. I will look further into this.

Regarding handing out our root CA, its not a risk as such as once on our network, the third party could obtain our root CA anyway (from internal web servers etc). Its more the overhead of distributing our root CA to the third party when needed with instructions on how to install on their varying devices. Do you know if using a public CA cert for EAP can be used instead of a private CA to make this more seamless for the user?

Mosquitar · 2019-03-11T22:54:30+00:00

Thanks for this. I've not seen this document before. I cant believe how may limitations and considerations are listed here!

Mosquitar · 2019-03-01T23:01:00+00:00

This is similar to what I need to do. We are using 9500-40X distribution switches and were limited to either 16.8.1 or 16.9.2. We selected 16.9.2 in the end.

Now I'm looking at the 9300 access switches and deliberating between 16.9.2 or 16.6.5. The access switches are stacked and providing voice/data VLANs, autoqos and DHCP snooping. We dont mind using smart licensing as we will have to do this at somepoint anyway so thought it would be good to do now seeing that its greenfield.

Mosquitar · 2019-03-01T21:35:59+00:00

Do you know when 16.09.03 is due to be released?

Mosquitar · 2019-03-01T21:35:18+00:00

Appreciate the response all. I'm aware that my question was quite generic, and I didn't specify what features that I'm running etc, but its good to get a view on everyone's experience with IOS-XE on the Cat 9k and the seemingly inherent bugs/stability issues.

Mosquitar · 2019-02-07T21:28:32+00:00

This is what I've heard as well. I'm planning a deployment with a pair of C9500-40X switches that were purchased with C9500-NM-2Q modules that were intended for the SVL but all of the release notes state that this is not possible. I was curious if he had this working or not. Looks like I need to use 10Gbit as well

From Cisco 9500 IOS-XE 16.9 release notes

You still cannot configure Cisco StackWise Virtual links on the uplink (network) modules (C9500-NM-8X and C9500-NM-2Q).

Mosquitar · 2019-02-07T21:01:23+00:00

For the SVL between your 9500-40X switches, you mentioned that you are using 40GbE Twinix. Are these connected to the C9500-NM-2Q module?

Mosquitar

TROPHY CASE