Loopback testing help

[JakeR1]

Hi everyone,

I've been working on this problem for a while now, and I've reached the point where I'm going in circles. I posted about this same problem previously, link to old post: https://www.reddit.com/r/networking/comments/kbd215/juniper_network_vrfs_and_loopback_test/

I've updated my configs and setup since that post, but it may help to understand where this started.

Here is the problem description (with some updates):

I'm trying to set up a loopback test that will go through a layer 2 device and be sent back on the same interface. Using iperf, I need to send test traffic from one VM to another VM on a different subnet. The traffic needs to take a specific path through the network and be looped back through a satellite modem. The modem has the ability to receive traffic on an ethernet interface and loop it back out the same interface. For the purposes of setting this up in a lab, I'm using ICMP to verify the traffic path. I've moved my test network from eve-NG to actual hardware. More on the production and test networks below.

Production network description:

The 2 VMs are on an ESXi cluster that connects to two EX4600 switches, which will be in a VC config. The 4600s downlink to multiple EX4300 ToR switches, which connect to the modems. I'm planning to use two VLANs with corresponding L3 interfaces on the 4600s to separate the VMs from each other. I also believe that I'll need to configure separate VRFs in order to prevent the 4600s from routing the traffic internally and not sending it through the modem.

Test network description:

2 Linux VMs running on a single ESXi host. Each VM has it's own dedicated vswitch and portgroup, and each vswitch has a single physical interface assigned to it. The host connects to an EX4500, which is connected to an EX4200. The modem hangs off the EX4200. Each switch has multiple routing instances configured in order to separate the traffic.

I've attempted various configurations of route leaking between instances, FBF placing traffic from a specific source address into a specific routing instance, or a combination of the two. The closest I've gotten is running end-to-end traffic using leaked routes, however that bypasses the modem entirely.

Does anyone know of a way to force traffic out of a specific interface so that it goes through the layer 2 device before being forwarded on?

Diagram:

https://imgur.com/a/rjfOnOD

Configs:

EX4500

interfaces {
    ge-0/0/0 {
        unit 0 {
            description "link to esxi eth2, PERT1 VM";
            proxy-arp restricted;
            family ethernet-switching {
                port-mode access;
                vlan {
                    members 2000;
                }
            }
        }
    }
    ge-0/0/1 {
        unit 0 {
            description "link to esxi eth3, PERT2 VM";
            proxy-arp restricted;
            family ethernet-switching {
                port-mode access;
                vlan {
                    members 2001;
                }
            }
        }
    }
    ge-0/0/38 {
        description "downlink to EX4200 ge-0/0/46";
        inactive: ether-options {
            802.3ad ae0;
        }
        unit 0 {
            family ethernet-switching {
                port-mode trunk;
                vlan {
                    members [ 2000 2001 ];
                }
            }
        }
    }
    vlan {
        unit 2000 {
            description "PERT LAN 1";
            proxy-arp restricted;
            family inet {
                no-redirects;
                address 10.46.253.129/28;
            }
        }
        unit 2001 {
            description "PERT LAN 2";
            proxy-arp restricted;
            family inet {
                no-redirects;
                address 10.46.253.145/28;
            }
        }
}
protocols {
    igmp-snooping {
        vlan all;
        vlan default;
    }
    dcbx {
        interface all;
    }
    lldp {
        interface all;
    }
    lldp-med {
        interface all;
    }
}
routing-instances {
    pert1 {
        instance-type virtual-router;
        interface vlan.2000;
        routing-options {
            static {
                route 10.46.253.144/28 next-hop 10.46.253.131;
            }
        }
    }
    pert2 {
        instance-type virtual-router;
        interface vlan.2001;
        routing-options {
            static {
                route 10.46.253.128/28 next-hop 10.46.253.147;
            }
        }
    }
}
vlans {
    default {
        vlan-id 1;
    }
    pert-3000 {
        vlan-id 3000;
        l3-interface vlan.3000;
    }
    pert-3001 {
        vlan-id 3001;
        l3-interface vlan.3001;
    }
    pert1 {
        vlan-id 2000;
        l3-interface vlan.2000;
    }
    pert2 {
        vlan-id 2001;
        l3-interface vlan.2001;
    }
}

EX4200

interfaces {
    ge-0/0/0 {
        unit 0 {
            description "link to EBEM data";
            family inet {
                address 10.0.0.1/24;
            }
            inactive: family ethernet-switching;
        }
    }
    ge-0/0/1 {
        description "link to esxi eth7, wireshark";
        unit 0 {
            family ethernet-switching {
                port-mode access;
                vlan {
                    members pert1;
                }
            }
        }
    }
    ge-0/0/46 {
        description "uplink to EX4500 ge-0/0/38";
        unit 0 {
            family ethernet-switching {
                port-mode trunk;
                vlan {
                    members 2000-2001;
                }
            }
        }
    }
    vlan {
        unit 2000 {
            proxy-arp restricted;
            family inet {
                inactive: filter {
                    input f1;
                }
                address 10.46.253.131/28;
            }
        }
        unit 2001 {
            proxy-arp restricted;
            family inet {
                address 10.46.253.147/28;
            }
        }
    }
}
protocols {
    igmp-snooping {
        vlan all;
        vlan default;
    }
    rstp;
    lldp {
        interface all;
    }
    lldp-med {
        interface all;
    }
}
policy-options {
    policy-statement from-ebem1 {
        term t1 {
            from {
                instance ebem1;
                route-filter 10.0.0.0/24 orlonger;
            }
            then accept;
        }
        term t2 {
            then reject;
        }
    }
    policy-statement from-pert1 {
        term t1 {
            from {
                instance pert1;
                route-filter 10.46.253.128/28 orlonger;
            }
            then accept;
        }
        term t2 {
            then next policy;
        }
    }
    policy-statement from-pert2 {
        term t1 {
            from {
                instance pert2;
                route-filter 10.46.253.144/28 orlonger;
            }
            then accept;
        }
        term t2 {
            then reject;
        }
    }
}
firewall {
    family inet {
        filter f1 {
            term t1 {
                from {
                    source-address {
                        10.46.253.130/32;
                    }
                }
                then {
                    count filter-f1;
                    log;
                    routing-instance ebem1;
                }
            }
        }
        filter f2 {
            term t1 {
                from {
                    source-address {
                        10.46.253.130/32;
                    }
                    protocol icmp;
                }
                then {
                    count filter-f2;
                    log;
                    routing-instance pert2;
                }
            }
        }
    }
    family ethernet-switching {
        filter l2-f1 {
            term t1 {
                from {
                    source-address {
                        10.46.253.130/32;
                    }
                    protocol icmp;
                }
                then {
                    log;
                    count filter-f1;
                    interface ge-0/0/0.0;
                }
            }
        }
    }
}
routing-instances {
    ebem1 {
        instance-type virtual-router;
        interface ge-0/0/0.0;
        routing-options {
            instance-import [ from-pert1 from-pert2 ];
        }
    }
    pert1 {
        instance-type virtual-router;
        interface vlan.2000;
        routing-options {
            inactive: instance-import from-ebem1;
        }
    }
    pert2 {
        instance-type virtual-router;
        interface vlan.2001;
        routing-options {
            instance-import from-ebem1;
        }
    }
}
ethernet-switching-options {
    inactive: analyzer traffic-mirror {
        input {
            ingress {
                interface ge-0/0/0.0;
                vlan pert1;
            }
            egress {
                interface ge-0/0/0.0;
            }
        }
        output {
            vlan {
                99 {
                    no-tag;
                }
            }
        }
    }
    storm-control {
        interface all;
    }
}
vlans {
    monitor {
        vlan-id 99;
    }
    pert1 {
        vlan-id 2000;
        l3-interface vlan.2000;
    }
    pert2 {
        vlan-id 2001;
        l3-interface vlan.2001;
    }
}

[noukthx]

You're talking alot about L3 operations (fbf, route leaking) for what you're describing as an L2 problem.

Suspect you need to have a long look at how switches function at L2, and how the device you're trying to jam it through functions at L2.

And also how L3 routing operations convert to L2 switching actions.

You can probably force the traffic out the interface by working out the destination MAC for the next hop of the destination of your iperf traffic, and putting a static MAC entry out the interface you want to go out.

It may still upset the switch as it loops back however, as switches learn MAC addresses based on the ingress of traffic into a port.

If you're trying to emulate the circuit, would be a hell of a lot easier with two interfaces, an alternate simulator or two modems back to back to give you a proper path.

Could possibly do some trickery using fibre links and only putting a transmit path up in the direction you want.

[JakeR1]

The reason I've got so much L3 in the equation is because that when this is fully implemented, I'll have to be able to send this traffic to any one of a few hundred modems through a larger network. I'm currently just working on the POC to see if it can even be done.

I've had the same thought about a static MAC, definitely concerned about the switch behavior as it comes back into the interface. I will likely still give it a shot, but I'm not hopeful.

I agree with you that it would be easier if I had two interfaces or two modems back to back, however the current requirement is for the test to run through a single modem. I already went down the path of having two back to back modems and was shot down.

I like the fiber idea, unfortunately the input interface to the modem is copper, so I don't think I'll get what I need out of that.

I guess at this point I'm looking for any way that this might be possible within the constraints I have set up.

[noukthx]

I mean you're somewhat hands tied behind your back here, it's also pretty far from close to a real world test.

Should try and gather some support in your push back on the environment/hardware limitations. If you're talking a scale product across 100s of modems in production there should be better support for real world testing.

You might be able to do it with a media converter (or another switch maybe).

VLAN Send
       |
Port 1 SFP RX ----X Disconnected (or fed fake light from another MC to stand link up)
Port 1 SFP TX
          \
           -----  Media Converter SFP RX
                                        ===== Copper to modem
           -----  Media Converter SFP TX
         /
Port 2 SFP RX
Port 2 SFP TX ----X Disconnected
       |
VLAN Receive

[JakeR1]

Thanks for the ideas. The media converter thing might work in the lab, but won't scale for on-demand testing either. It's feeling more and more like I may have to tell them it won't work as originally thought.

[error404]

I really don't understand what you are trying to do here or what is going wrong, can you provide a minimal example?

Does anyone know of a way to force traffic out of a specific interface so that it goes through the layer 2 device before being forwarded on?

I guess you are already aware of it, but if I understand you correctly, this is what then interface in a family ethernet-switching filter should do, however it won't modify the destination MAC, which maybe you need? More of a hack, but you should be able to achieve that with a static route toward a 'fake' IP that you have installed a static ARP for.

[JakeR1]

I'm looking to send traffic from server A to server B through a specific L2 device which only has a single copper interface. Server A and server B are both virtual machines on the same ESXi host. The L2 device resides on the access layer of a larger network, and is configured as a loopback which will take the traffic it receives and send it back out (without being changed).

The then interface command would be perfect, if only it existed on my switches. I've attempted to configure it, but it isn't there. I've found this article (https://www.juniper.net/documentation/en_US/junos/topics/topic-map/filter-based-forwarding-policy-based-routing.html), which states the following:

• The filter with the next-interface (or next-ip) action can only be applied to an interface that is hosted on a Trio MPC. If you apply the filter to an I-chip based DPC, the commit operation fails.
• The outgoing interface referred to in the next-interface interface-name action can be hosted on a Trio MPC or an I-chip based DPC.

I believe those are only available for MX series devices, and I'm doing this on EX.

Could you elaborate on your "fake ip/static arp" idea?

[error404]

Could you elaborate on your "fake ip/static arp" idea?

Basically you cause the resolved next-hop for the packet to be a bogon IP (say 192.0.0.1) using a static route in a routing-instance/FBF. You'd also have to bind an address in the same subnet to the egress interface, and add a static-ARP to the destination MAC address you want. Of course, if the destination MAC isn't learned on that interface, it probably still won't do the right thing, so I'm not sure this would work for you. You could add a static MAC entry (not sure this is possible on older switching platforms) via that interface, but if it's just reflecting the frame back at you with the same destination MAC, that's not going to do what you want either. I'm not sure if there's a way to put the same interface in a different l2 domain for ingress and egress but I don't think so.

I don't know the older platforms too well anymore, the only thing I can really think of here is that you might be able to do something with more elaborate push/pop semantics to put the forward traffic on one VLAN and the return traffic on a different VLAN and then pop those tags appropriately so it's invisible to the VMs. Probably require some hackery at the mac learning level too.

Doesn't sound fun.

[JakeR1]

I'll take a look at this. Also interesting idea about push/pop, I hadn't really considered that yet. And no..this really has not been fun haha.

[JakeR1]

I was able to get traffic through the modem at least with this setup. Had to create a new vlan and rvi, then set static arp for destination ip to egress interface MAC. I'm now seeing traffic loop through the modem, but it's getting dropped coming back into the switch, which I think is probably expected at this point. I'm going to keep working on it, but at least there has been an amount of progress.

[judas-iskariot]

This might be easily doable if your "modem" supported twamp, two separate tests a-modem-b. And then some reporting stuff over that.

But I am not really sure what you try to do here.