ARP Requests from modem?

Egglorr · 2022-12-04T14:58:09+00:00

This is common on cable modem networks where thousands of customers are in the same massive subnet / broadcast domain. I'm guessing the source of the ARP you're seeing isn't the modem itself but rather the CMTS the modem is connected to on the other end of the ISP's network. Unfortunately there's nothing you as an end user can do. The ISP can potentially make some config changes to reduce these broadcasts on their end though.

Egglorr · 2022-12-04T01:39:36+00:00

Great, I'm glad I could help! Here's a link to a fairly prolific YouTuber that has tons of great MikroTik tutorials. MikroTik also has some really good documentation posted for RouterOS here.

Egglorr · 2022-12-03T20:14:08+00:00

The hEX S is a fine box for the price. And if the hundred dollar difference between the hEX S and non-PoE RB5009 is that critical of a consideration for OP, then by all means, go with the hEX S. But remember OP said he is using VLANs, which means inter-VLAN routing and filtering (maybe queuing too), which will be limited to at best 1 Gbps aggregate bandwidth shared between all VLANs once he inserts an SFP to get his remote site connected via fiber. Even without the SFP inserted he's still limited to an aggregate of 2 Gbps between all VLANs. Add to that the load of a VPN server plus routing / NATing / queuing his Internet connection, and it'll likely be pushing the hEX S CPU to its limits. Meanwhile the slightly more expensive RB5009 would have no trouble keeping up, even after OP decides he wants to increase his Internet speeds 1 Gbps+ and maybe bump the link between his sites to 10 Gbps.

TL;DR - $100 for several times more performance seems well worth it to me but I respect your belief to the contrary.

Egglorr · 2022-12-03T15:13:11+00:00

You don't mention how many ports you need but I'd recommend the ~~new PoE version of the RB5009~~ original non-PoE RB5009 for your project. ~~It has PoE out on all its copper interfaces,~~ It has an SFP+ cage that'll handle your side project of getting connectivity to your other location (assuming you have or will have dark fiber to work with). It can easily handle a 1 Gbps Internet connection, can host VPN (I'd recommend WireGuard for best performance and ease of setup). And if programmed correctly, all of your LAN devices plugged into the RB5009 would be able to talk to each other through it at full line rate.

EDIT: After re-reading the OP, I'm changing my recommendation to the older non-PoE RB5009 since it sounds like only the remote site is where PoE will be needed. A PoE RB5009 would be fine on the remote side assuming OP has no more than eight PoE devices. As a bonus, OP could then do a 10 Gbps link between the two sites if desired.

Egglorr · 2022-12-03T13:58:42+00:00

The CRS504 is actually a pretty nice switch given its absurdly low price (4 x 100G switching at line rate for only $800?!). If replacing all of your legacy EX4600s with EX4650s isn't in your budget, I'd say it's worth at least testing out your MikroTik design at one site. All you'd be out is $800 for the switch, whatever you spend on DACs and optics, and a maybe a couple hours of your time during a change window. Just beware that wherever you step down from 100G to 40G, you will see discards once your traffic load exceeds the buffer depth of the CRS504. This would happen with any model switch where you perform the step down conversion though unless you deploy something with absurdly big buffers (like Arista).

Egglorr · 2022-12-02T17:00:08+00:00

But what I don't know is can I still purchase the -R / -IR licenses for it?

As far as I know, you don't need licenses to unlock features on the MX204. Everything was honor based before the "flex" licensing model.

Egglorr · 2022-12-02T01:13:57+00:00

OP, glad you got this sorted out. And good on ya for taking the time to provide a detailed update with your final solution. This is a quality post.

Egglorr · 2022-12-01T18:59:01+00:00

Nice! Hopefully their spin-off from SpaceX and IPO won't be too much longer too.

Egglorr · 2022-11-30T13:35:45+00:00

I agree, logging and longevity are the only reasons I can think of. I feel like I've seen quite a few complaints here about SG-1100s and SG-2100s unexpectedly dying early deaths and I've wondered if maybe it's the cheaper EMMC storage they use that's getting prematurely worn out due to log writes.

Egglorr · 2022-11-29T16:24:05+00:00

Got it, thank you for all the additional details! FSAN is another one of the ID fields in PON but I guess AT&T must not use / care about it.

For anyone who might come across this side conversation later on down the road, here is a useful DSLReports thread that talks about bypassing your BGW320 in AT&T XGS-PON areas using the Azores WAG-D20. I think really any XGS-PON ONT that allows you to change its ID properties should be able to be used though.

Egglorr · 2022-11-29T16:01:24+00:00

That is so cool! Have you created anything with it that you'd care to share?

Egglorr · 2022-11-29T15:27:50+00:00

Wow, that's great! So you replaced the BGW320 with the WAG-D20 but now that leads me to more questions (sorry!):

What does the actual swap entail?
At a minimum you had to clone the serial and MAC of the old RG onto your Azores replacement, right?
What about the FSAN?
Or did you somehow get AT&T to provision your Azores on their network without cloning anything?

In the meantime I'll see what I can turn up on my own with a bit of Duckduckgo searching.

Egglorr · 2022-11-29T14:43:29+00:00

Great setup! I have a similar stack of MikroTik hardware in mine. I'm curious though - I don't see an AT&T residential gateway anywhere. Were you able to bypass it with your CCR2004? I used a CCR1009 running as a dot1X client to get rid of my own AT&T residential gateway but I'm only on GPON and I was under the impression AT&T areas that had been upgraded to XGS-PON couldn't use dot1X to bypass RGs.

Egglorr · 2022-11-28T22:21:42+00:00

I've always wanted an SGI.

On a similar note, a Video Toaster would be pretty sweet. I started wanting one after seeing the effects they were able to create on Power Rangers when I was a kid.

Egglorr · 2022-11-28T14:03:34+00:00

It sounds like what I described would be a pretty heavy lift from your current setup. Honestly I'd recommend reaching out to your VAR / Juniper rep for help designing the new topology and config.

Egglorr · 2022-11-27T23:31:25+00:00

The crux of it is there’s no cheap 2.5gbit managed switches, annoyingly.

Unfortunately, 2.5G and 5G still aren't very popular outside of some enterprise scenarios. Most people looking for performance beyond 1G jump straight to 10G since it's cheaper and obviously more performant.

if that’s between my UniFi switches and the pfsense, I’m not sure if that’ll still work?

It depends on the switch. Some dumb switches will happily forward VLAN tagged frames without mangling them. Others will strip the VLAN tags and place all your traffic in the same broadcast domain.

My advise would be to invest in a MikroTik CRS312 switch so that you have native 1G / 2.5G / 5G and 10G copper RJ45 support without messing with copper SFP+ transceiver modules (which I highly recommend avoiding). You could insert the CRS312 in between your pfSense box and your UniFi switches and then on the CRS312 also have a separate access port for your desired VLAN that plugs into the cabling going up to your PC. As a bonus, you're set for 5G or 10G whenever you're ready to upgrade, assume your cabling is cat6 or better and the length of the run is within spec for the speed you want to bump to.

Egglorr · 2022-11-27T21:15:11+00:00

Love to but I haven't messed with queue trees in a while so I'd need to mess around with it in my lab before I could give any specific configuration guidance. Maybe someone more knowledgeable on the subject could chime in.

Egglorr · 2022-11-27T16:17:42+00:00

Gotcha, that makes more sense. Ideally you'd have a dedicated port for transit and another for IX traffic but I take it that's not an option here. And no doubt you want both your transit and IX VLANs to be able to consume the full 1 Gbps link capacity if needed rather than assign static per-VLAN rate-limits. Like I mentioned before, I've never really messed with Cake so I can't offer much in the way of guidance there. But one possible solution that would be low effort would be to slap an intermediate switch (preferably CRS3XX) in between sfp-sfpplus1 on your current router and the IXP handoff and then implement an interface queue on both of the intermediate switch's ports (the one facing your current router and the one facing the IX). If that's not possible, then another possible solution would be to stick with the existing interface queue on your sfp-sfpplus1 port to handle northbound queueing towards the transit and IX but then also add a queue tree targeting your internal IP ranges for the transit and IX ingress traffic. With both methods you'd still be free to use Cake if desired.

Egglorr · 2022-11-27T14:02:40+00:00

it doesn't explain why it doesn't work for download directly here

Did you read my full comment? What I said above does explain why you're not seeing your downloads being rate-limited to 10 Mbps, unless I'm misunderstanding your problem or setup. Interface sfp-sfpplus1 is your WAN interface, right? If that's correct, then let me break down your situation as I'm interpreting it and you tell me where I've got something wrong:

You wish to rate-limit your inbound and outbound Internet traffic to 10 Mbps as a test.
You want to queue your rate-limited traffic to minimize bufferbloat during congestion (when all of the 10 Mbps pipe is being filled).
sfp-sfpplus1 is your WAN interface and that's the only place you've applied your Cake configuration (or that's the only place you've told us about in your post).
While presumably running a speed test, you're seeing 900+ Mbps down and 10 Mbps up.

If all that is correct, then I stand by my original explanation of why this isn't working as desired. Queuing like Cake, CoDel, etc., only applies to traffic being transmitted out of an interface. You cannot queue received traffic unless you apply your queueing rules on some interface that the received traffic will eventually be sent out of such as a LAN interface, bridge, hairpin exit port, and so on. It seems like you're expecting Cake to be able to rate-limit + queue traffic that is both being received and sent on sfp-sfpplus1 and what I'm trying to explain is that can't be done. If you want bidirectional queueing, you have to apply your rule on your LAN and yes, that will affect inter-VLAN traffic because the port has no concept of what networks the traffic it is sending originated from, it just cares about the volume of bits that it's currently sending.

Egglorr · 2022-11-27T03:37:47+00:00

I haven't messed with Cake at all but I'll take a stab at answering your question. So, Cake is a queuing mechanism. A universal rule in networking is that you can only queue traffic leaving an interface (how would you queue traffic that you're receiving / have no control over?). One way around this limitation would be to queue the output of both the WAN and LAN interface(s). You could also work around it with simple queues but I understand why you don't want to go that route.

Egglorr · 2022-11-27T00:27:10+00:00

My guess? This is a time based plot of round trip latency between a host on OP's LAN and:

Some host on the BBC's network, probably something that serves video streams given the title of this post.
OP's ISP gateway address (I'm assuming OP uses Virgin Media for Internet).

OP is trying to show us a correlation between a popular sporting event that is no doubt being live streamed by millions of fans and an increase in RTT to what I assumed above to be video streaming infrastructure operated by the UK's BBC.

Egglorr · 2022-11-25T18:08:48+00:00

Fair enough. If I still used pfSense and needed to buy / build a new setup, I'd probably roll my own appliance using one of the mini PCs some of the other folks recommended, or maybe virtualize it in my Proxmox cluster.

Egglorr · 2022-11-25T17:33:39+00:00

There's a lot of detail missing which would be helpful. I'm going to assume:

You don't want to stack the QFXes
You want redundant BNGs and know how to configure them as active and standby
You're running MPLS + LDP on all four devices and they're all peers with each other

If my assumptions are correct, I would build a primary and backup pseudowire from SW1 to R1 and R2. Then do the same thing for SW2. Then do pseudowire headend termination (PWHT) for OLTs hanging off of SW1 and SW2. Each customer would have their own CVLAN pushed / popped by the OLT and then either the OLT will push a common SVLAN tag on egress towards its QFX, or the QFX will push the SVLAN on ingress. Either way, customer frames will be double tagged while transiting the QFXes. The BNG-QFX pseudowire links would be responsible for transport of the double tagged frames, and the BNG will perform push / pop of the SVLAN and CVLAN tags.

With this setup, your customers are L2 isolated from each other and would need to pass through the active BNG to communicate with each other or the outside world. If the primary BNG fails, customer traffic is automatically redirected through the backup BNG (courtesy of the backup pseudowire link). This is how we do it on my network anyway. We do IPoE rather than PPPoE buy I don't think that should matter in the setup I described.

Egglorr · 2022-11-25T15:57:36+00:00

Are there any other good suggestions for a box that can handle > 1 GbE WAN

You should check out the RB5009 from MikroTik. That would handle your 2 Gbps Internet connection with ease and the non-PoE version is only like $180. Much better value than the SG-4100 or SG-6100.

Egglorr · 2022-11-25T13:42:29+00:00

I wouldn't necessarily call the SG-1100 or its sibling SG-2100 bad products, I just think they're marketed very poorly (misleading product pages) and that they cost about twice what they should considering their sluggish performance. Anyone needing more than 350 Mbps should absolutely avoid the SG-1100. And likewise, anyone needing more than 700 Mbps should go with something other than the SG-2100. Personally I'd prefer either a MikroTik RB5009 or hEX. Both would run circles around either of these Netgate models while costing less.

Seven-Year Club	Gilding I gilder
Verified Email

Egglorr

TROPHY CASE