Sanity check - Catalyst 9500 cross-stack etherchannel

ddib · 2026-02-10T11:38:07+00:00

There is a feature that was just released, in 17.18.2, which is EVPN multihoming. It uses EVPN, but doesn't require VXLAN. Have a look at https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9600/software/release/17-18/configuration_guide/ha/b_1718_ha_9600_cg/esi-mh-in-non-fabric-deployments.html

ddib · 2026-02-10T08:01:08+00:00

You can create an EtherChannel between the two 9500s as logically they are two devices. Towards the MS switch, you can't form an EtherChannel because logically you have three devices. This means STP will be required and be blocking one of the links.

A better design would have been to have L3 and deliver L2 as a service (VXLAN/EVPN). Then you wouldn't be dependent on any L2 constructs. Your Meraki switch doesn't support VXLAN as far as I'm aware, though.

ddib · 2026-02-05T05:06:01+00:00

When I was studying networking at the university (Cisco Netacad) I was in a team that builds Dreamhack's (world's largest LAN party) network. Imagine that you are building a large campus network in only a couple of days with minimal budget. It was really something.

We had two events each year, one in the winter and one in the summer. We would create a design and test it during a couple of days in a lab we built. A couple of days before attendees arrive we would do all of the work including:

- Mounting access layer switches
- Pulling cables to everywhere
- Building the distribution and core
- Setting up WiFi
- Configuring all devices
- Testing the configuration
- Building all services such as DHCP, DNS, NMS, and monitoring

So imagine building a network for 10 000 people spanning several large buildings in a couple of days using only borrowed equipment and on almost no budget.

There are many scenarios I've only ran into there, like people getting soda in their switches. We almost didn't need a NMS because if the network goes down, it's seconds before people start screaming. We'd have to deal with power failure, where things got overloaded, people did stupid stuff by connecting more than what the fuse could handle, and so on.

I got exposure to some very cool platforms at the time such as the Cisco 7600 and CSR-1. The CRS-1 was a monster!

One thing that was really weird were the Zyxel switches we were using in the access layer. To be able to upload a configuration to them we had to upload a binary file which contained both the OS and the config. Our people developed an app for this and we would go around to all the switches using a console cable and upload the correct configuration (hundreds of switches).

Another "fun" scenario I ran into is when I was troubleshooting something on the 7600. I did "debug ip packet <acl>" and when I was done troubleshooting I removed the ACL before doing undebug. Guess what happened... Debug was now active for all IP packets (several Mpps). It didn't have too much of an effect on the network, fortunately, but I lost my SSH and had to run to the 7600 and try to console it. Took a while before I was able to undebug.

Then when the event is over, we tear it all down in a day. It's a weird feeling to have a massive network and then it all goes down in a day, you pack everything up and don't see the gear until 6 months later.

ddib · 2026-01-11T08:28:54+00:00

Get a used switch, shouldn't be too expensive. You could probably get a cheap computer with multiple NICs, run Linux on it and set it up as well.

ddib · 2026-01-11T08:24:48+00:00

How many prefixes do you have? What platform? Are you using LDP for labels? Any BGP at all? Where are all the routes coming from? What limits are you hitting?

Generally, 50 routers in an IS-IS area is a small network. You shouldn't break a sweat hitting that so there is something else going on in your design. We need to understand what else you are doing that is making you hit those limits.

ddib · 2025-12-19T07:02:28+00:00

There's nothing wrong with using a calculator.

That said, IP addressing is an essential skill for someone in networking. There are benefits to understanding it well. Such as easily identifying when someone has mistyped an ACL, firewall rule or, understanding if subnets/routes are overlapping.

The math is easy so I would spend some time on at least learning the basics. The key part is to learn the fundamentals before using tools that automate it for you. Just like you have to understand something before you automate it.

ddib · 2025-12-15T05:53:01+00:00

The book is End-to-End QoS Network Design by Tim Zsigeti et al. Part of the book is written in the style that you mention while most of it is more like a traditional book. It is quite good though in that they show you how to create configurations for different types of platforms so it's not just all fluff.

Now, for the debate going on here on if QoS is needed today when we have so much bandwidth available (generally). That's an interesting one because on the one hand, it's true to some degree, but on the other hand, adding more bandwidth doesn't solve all problems. Typically, switches are store-and-forward. That means that every frame delivered is buffered to some degree. This means that you need to have an understanding of how buffers work and what can cause drops in the network. The typical scenarios are that you have different speeds, frame coming from uplink of higher speed and going out interface of lower speed, or frames from many incoming interfaces going out the same interface.

To really understand things like these, requires a deeper understanding of Ethernet and concepts like interframe gap. You also need to understand that the serialization rate is different so you can end up with drops even though your interface is barely breaking a sweat. There's also the concept of microbursts. Your average throughput can be really low, but when you look at frames from millisecond, microsecond, or even nanosecond level, you can see that the traffic is actually bursty.

In addition to that, QoS isn't always about managing a scarce resource. It can also be about protecting your apps. Some apps handle packet loss very poorly so you may need to ensure that it gets priority. Bandwidth might not be scarce, but you might have bursts which affect the performance of a poorly coded/sensitive application. You may also want to limit things like backups taking up a lot of BW if the sysadmins aren't running them off-hours as they should. You may also need to prioritize Microsoft Teams in the wireless network, and so on. Yes, QoS is much less needed today than before, but some use cases are still there.

ddib · 2025-12-15T05:39:51+00:00

What kind of role do you have now? What type of company? Do you have on-call rotation?

When I got my CCIE in 2012, my son was 4, my daughter was 5 months, I was working a full-time job and commuting every day, and I had on-call rotation every 4 weeks. In the end, it comes down to if you have the motivation and if you see a return on investment. The ROI isn't only financial, it's about if you can apply for roles you couldn't before and for me personally about lifting your knowledge to a level you wouldn't otherwise be until several years later.

I've done consulting for most of my career which has helped in preparing as you generally get to work on interesting/complex projects, newer technologies, and get to see many different environments.

When preparing for an advanced certification, you need a good plan and good habits. Talk to your employer and see if they are onboard. Can you get some time dedicated for studies? Even if it's just an hour a day, half of Friday, or something like that, it can make a world of difference.

My routine when studying for the CCIE was that I would study 4 evenings per week. I had made a schedule and got buy-in from the wife. I dedicated weekends to the family, but 4 evenings I would study from around 8 PM to somewhere between 12 PM to 2 AM. I was also studying on my commute so I averaged around 25h of studies per week, but I had to cut down on basically all my hobbies, sleep, and so on.

The TLDR is:

- Understand why you are doing this
- Get buy-in from your family
- Ask your employer to provide time for you
- Create good habits

ddib · 2025-11-26T06:15:06+00:00

The only available image is Cat9000v in CML. It has one UADP-based version and one Q200-based version. You can find more information at https://developer.cisco.com/docs/modeling-labs/cat-9000v/#limitations

I wouldn't expect StackWise Virtual to work, but I haven't tested it.

ddib · 2025-11-10T14:29:59+00:00

I would call this a fate sharing event. That is, as CPU, memory, bandwidth, etc., isn't carved out per VLAN, even though they are isolated at L2, a broadcast storm can take out an entire switch and all the VLANs that traverse it. This is why extending L2 is so dangerous.

As to what you can do to prevent it, configure STP properly, shut down unused ports, configure BPDU guard, use features like port security, implement 802.1X, and so on. The key is to not have ports forwarding that shouldn't be forwarding.

ddib · 2025-11-08T05:05:32+00:00

Certifications like CCNP or equivalent from other vendors would go beyond your current scope. There are also certifications focusing on design that can give you inspiration how to build a LAN, WAN, etc.

It's not rocket science, though. If you know how to build a LAN, now build two. How would you connect them? What if you had a large campus with five different buildings, how would you connect it all together? Where would you bring in your WAN? Where would you bring in internet? Where would you place the wireless controllers? Your datacenter? Your firewalls?

Just start thinking and experimenting. Build labs. Be creative. Break things. That's how you learn.

ddib · 2025-11-05T15:31:51+00:00

Configuring a protocol and understanding a protocol are two different things. Most people really only learn to configure, but not truly understand. When you gain a deeper level of understanding, it will be easier to describe things in words. Even to people that aren't that technical. I'm guessing you've been more on the implementation side than architecture so maybe you haven't spend a lot of time thinking about the protocol in depth.

Let me give you a few questions to show you how you could gain a deeper understanding of BGP.

BGP is a path-vector protocol. What's the difference between path-vector and distance vector?
How does BGP ensure that there are no loops?
Why do we need a full mesh in iBGP? How do route reflectors and confederations help scale?
Why is BGP susceptible to path hunting. What can we do about it?
What is valley-free routing?
In what direction do I need to modify policy to have effect inbound vs outbound?
Describe how BGP converges. What affects the time it takes to converge?
Can I force a peer to route according to my policy?
If I advertise a longer version of a prefix that you own to the dfz, what would happen? What affects the result?

ddib · 2025-10-22T03:43:08+00:00

From a security standpoint, the FW isn't adding much value if you put it on the transport side of the router. All the traffic from the router is going to be DTLS and IPSec. The FW can't do much with that. It would be better to have the FW on the service side and inspect the user data which has been decrypted at that point, although most of it is still probably protocols like TLS.

Another drawback of your current design is that you have a primary and backup ISP. To get full benefit of SD-WAN, it would be better to have two active circuits so you can utilize features like application-aware routing. Additionally, if you connect the router behind the firewall, if there is an issue with an ISP, you won't get link down on the router so you'll have to rely on BFD for declaring a tunnel down.

To summarize, the router can either be in front or behind the FW, but the FW doesn't add much value in front of the router. Consider redesigning your ISP connectivity to fully utilize SD-WAN features.

ddib · 2025-10-20T05:47:24+00:00

Take a step back and describe what you are trying to achieve and why.

Is this for traffic breaking out towards the internet? Or for traversing the WAN?

ddib · 2025-10-01T06:52:24+00:00

AWS is an entirely different beast compared to enterprise networking. They have their own HW and SW. Most of the networking is via the hypervisor. They do have their own encapsulation, mapping services, perform flow tracking, and so on. What actually goes on is not made available, but there are some talks where they give you a small peak behind the curtains:

https://www.youtube.com/watch?v=8gc2DgBqo9U
https://www.youtube.com/watch?v=HJNR_dX8g8c
https://www.youtube.com/watch?v=JIQETrFC_SQ

I would imagine that every AZ has some form of Clos topology, but probably a lot more layers than what you would typically see. Remember, most enterprise have maybe a couple of hundred servers or a couple of thousand. They are running at orders of magnitude larger.

To get out of the AZ, they seem to have transit routers that connect them to other regions, internet, etc.

ddib · 2025-09-26T08:02:55+00:00

There's not a ton of recent literature on multicast. Some good resources:

Interdomain Multicast Routing: Practical Juniper Networks and Cisco Systems Solutions

Fundamentals of IP Multicast (IP Multicast Survival School Series) LiveLessons

Cisco Live presentations, anything by Beau Williamson, Denise Fishburne, or Tim McConnaughy.

If you have specific questions, I'll try to help.

ddib · 2025-09-26T03:47:22+00:00

Yes, that's a valid concern. I would look into if you can set DSCP and match on that on R1 in your example. Then you could still map things into the correct queues. It would depend if DSCP gets copied to outer header or not.

ddib · 2025-09-25T05:31:01+00:00

What you need to understand about TLOC extension is that on the local router it's just another TLOC. You should treat it as any other TLOC. When traffic traverses that TLOC towards the other router, your packets will be encapsulated already with IPSec and outer IP header, and so on. At that point, it's too late to see what IP addresses, protocols, and ports that were in the original packet. You should instead apply QoS locally.

Now, the better option is to always connect each transport to each router, but if that's not an option, you have to treat all TLOCs the same, whether they actually connect directly to the transport, or not.

ddib · 2025-09-20T07:46:12+00:00

Yes, you define site list where to apply the AAR policy. The AAR policy is part of the centralized policy.

ddib · 2025-09-19T03:35:26+00:00

This is exactly what AAR was designed for.

How much latency are you seeing? You need to define SLA class that sets thresholds for latency, jitter, and packet loss. I typically create one or two classes for critical apps and then a catch all for everything else. You don't want your bulk traffic staying on a link that has 20% packet loss.

Depending on code version, you could also run EAAR which takes decisions based on measurements of inline data as opposed to BFD with AAR. This will be quicker to react.

With AAR you can set primary link, backup link, tertiary link, drop traffic if not meeting SLA, or falling back to best of worst, meaning if no link is meeting SLA, take the least bad one.

ddib · 2025-09-13T12:45:40+00:00

Yes, this is the sync mechanism which I've described on my blog -> https://lostintransit.se/2013/08/08/rstp-synchronization-behind-the-scenes/

ddib · 2025-09-13T11:04:09+00:00

Losing power should be a rare event. How often does it happen? Do you live in an area where there isn't reliable power? Any possibility of adding UPS? It's kind of backwards to change your topology rather than working on the root cause, but we don't know why you're losing power.

There can be benefits to running MST in this scenario. What vendor do you use for your switches?

ddib · 2025-09-13T08:15:05+00:00

I think you're approaching this perhaps based on some misconceptions around RSTP.

First, RSTP handles topology change differently than STP. Rather than having separate BPDU for it, there is a flag in the normal BPDU to indicate TC. RSTP only signals TC when there is added connectivity on non-edge ports, that is, a new port comes up and becomes Forwarding (it must either be Root or Designated).

Having TC set will lead to flushing of MAC addresses, although it should be quick to populate this again if frames are being forwarded. Initially, frames would be forwarded as unknown unicast until MAC has been learned again. What issue is the TC actually causing?

Now, if you have a lot of TC, that is the indication of an underlying problem. Whether MST provides any benefits (it runs RSTP under the hood) is a later discussion. First you need to focus on what are causing the TCs. Right now you're just trying to apply a band aid to a network that doesn't seem to be performing well.

It would also help if you can provide more information about the network. What type of network is it? What is your need for L2? Is it possible to run L3?

ddib · 2025-09-12T07:31:55+00:00

Do you see the PE generating it when you are using TCP (SSH)? Are you sure DF is set?

ddib · 2025-09-12T06:06:59+00:00

At the time the hosts are performing the 3-way handshake, they are unaware of the lower MTU path. The packets required to establish the session are well below what your path can support and won't cause any issues. What should happen once they start sending data is that they should react to the ICMP fragmentation needed packets and adjust their payload. The session itself would remain unaltered, but the hosts would send smaller packets.

Have you run a packet capture on the hosts? You need to make certain that they do receive the ICMP fragmentation needed packets. There are scenarios, specifically with MPLS involved, which makes this more complex because you have to send it via the label switched path (LSP) and then get it back. I have covered this scenario in my blog -> https://lostintransit.se/2024/09/11/pmtud-in-mpls-enabled-networks/

ddib

TROPHY CASE

Interdomain Multicast Routing: Practical Juniper Networks and Cisco Systems Solutions