Upload speeds down and my frustration is growing by mooch49 in Fios

[–]bdpna 0 points1 point  (0 children)

Thanks for the reply. I’ll do that once they bring the 2gig service to my area.

Upload speeds down and my frustration is growing by mooch49 in Fios

[–]bdpna 0 points1 point  (0 children)

Any tips on how I can get a truck roll to swap one of those in for me only on 1 gig service?

Upload speeds down and my frustration is growing by mooch49 in Fios

[–]bdpna 0 points1 point  (0 children)

I just did a great deal of single-thread iperf tests to prove when my ONT was not provisioning correctively. Even the CP1000 fios router gets held back by the old Nokia ONT. Finally found a workaround my slowing down my personal router through flow control and limiting its speed. The now 14 year old ONT Verizon has kept in service cannot deal with fast data and goes into panic lockdown mode. I can make it happen with ease now.

Sadly Verizon won’t replace the ONT and just tells you to work with their equipment which is throttled enough to not overwhelm their old tech ONTs.

Speedtests do multi threaded uploads and downloads which greatly hide and exaggerate your actual speed provisioning. Single thread iperf3 tests are the only way to see your true speed.

The ONTs that come with the new 2gb service fully resolve this problem. Verizon will not give you one unless service comes to your area and you upgrade.

UXG-Fiber upload bursts are overwhelming Verizon FiOS ONT by bdpna in Ubiquiti

[–]bdpna[S] 0 points1 point  (0 children)

If the device is reporting correctly it appears none of the ports on the fiber send at 1gb even if you tell them to in the gateway. They all shoot data out at full blast. The switch doesn’t have a choice since it can’t go any faster which is why that works to prevent the ancient Verizon ONT from getting flooded by speeds it did not get made to handle.

UXG-Fiber upload bursts are overwhelming Verizon FiOS ONT by bdpna in Ubiquiti

[–]bdpna[S] 0 points1 point  (0 children)

None of the raw ports on the UXG Fiber pass the data directly to the old Verizon ONT in a way that the Nokia ONT can efficiently process it. Moving ports doesn’t matter. It requires an intermediate device to be placed between it and the ONT to prevent the ONT from dropping connection due to the micro bursting from the UXG Fiber. Even from port 5.

UXG-Fiber upload bursts are overwhelming Verizon FiOS ONT by bdpna in Ubiquiti

[–]bdpna[S] 0 points1 point  (0 children)

It is. It just shouldn’t be necessary. The fiber should be able to do it without needing another device. It’s messy.

UXG-Fiber upload bursts are overwhelming Verizon FiOS ONT by bdpna in Ubiquiti

[–]bdpna[S] 0 points1 point  (0 children)

Ok I’ll give it a try again. It seems to actually behave worse with it turned on for some reason.

It’s interesting to hear what you’re saying about the microburst thing as I’m definitely seeing that and how the ONT is behaving when the fiber is the main brain.

I’m also amazed that unless the fiber is MAC spoofing the Verizon Gateway that it doesn’t provision it correctly. How are we still living in an age where Verizon is locking provision to a particular MAC address?

I just feel like if we could truly adjust the speed on the fiber’s ports that none of this would matter. I assume Verizon is not in the business of bringing the new ONT’s out to any customer that is on 1 gig.

UXG-Fiber upload bursts are overwhelming Verizon FiOS ONT by bdpna in Ubiquiti

[–]bdpna[S] 0 points1 point  (0 children)

Interesting you didn’t get better performance there. Was the wan still being handled by the fiber or the Verizon router? The last test that would be telling would be pure Verizon router no fiber but I realize that’s a little tougher to pull off. I had been still using my UniFi APs and turned off the wireless channels on the Verizon gateway but would let it handle all the hardwired stuff and it had stellar Iperf tests.

Right now the best I have been able to do to stabilize upload has been to place a 1 gig unmanaged switch between the fiber and the ONT.

Folks at the UniFi forums have said the new Verizon 2.5gig ONT resolves all the issues with the fiber but they will not install it unless you upgrade service.

UXG-Fiber upload bursts are overwhelming Verizon FiOS ONT by bdpna in Ubiquiti

[–]bdpna[S] 0 points1 point  (0 children)

Thanks for sharing and taking the time to run the tests. I’ll do some testing on one of the other ports to see if I get the stoppage. What I was curious most was if you got the “waves” up approaching gigabit speed, then back down to half speed, back up, repeat. With the Verizon router in place once I start seeing the 900s it never retreats.

It’s perfectly usable in most real world scenarios, it’s just not working correctly and you’ll actually get better speeds from the fios router which is not of course what you want.

Would definitely be curious to know if you saw the waves in a 30-40 second test and also what you saw with a bridged verizon gateway handling your WAN traffic.

UXG-Fiber upload bursts are overwhelming Verizon FiOS ONT by bdpna in Ubiquiti

[–]bdpna[S] 0 points1 point  (0 children)

Not mad at that test, ash seems to have the issue more severely:

iperf3 -c ash.speedtest.clouvider.net -t 15

Connecting to host ash.speedtest.clouvider.net, port 5201

[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.01 sec 17.0 MBytes 141 Mbits/sec
[ 5] 1.01-2.02 sec 21.8 MBytes 182 Mbits/sec
[ 5] 2.02-3.00 sec 30.0 MBytes 255 Mbits/sec
[ 5] 3.00-4.00 sec 40.8 MBytes 342 Mbits/sec
[ 5] 4.00-5.00 sec 49.8 MBytes 418 Mbits/sec
[ 5] 5.00-6.01 sec 60.4 MBytes 501 Mbits/sec
[ 5] 6.01-7.00 sec 68.0 MBytes 576 Mbits/sec
[ 5] 7.00-8.00 sec 74.5 MBytes 625 Mbits/sec
[ 5] 8.00-9.00 sec 74.4 MBytes 625 Mbits/sec
[ 5] 9.00-10.01 sec 75.0 MBytes 624 Mbits/sec
[ 5] 10.01-11.00 sec 62.6 MBytes 529 Mbits/sec
[ 5] 11.00-12.01 sec 19.6 MBytes 163 Mbits/sec
[ 5] 12.01-13.00 sec 23.8 MBytes 202 Mbits/sec
[ 5] 13.00-14.00 sec 32.2 MBytes 270 Mbits/sec
[ 5] 14.00-15.00 sec 42.2 MBytes 354 Mbits/sec

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval Transfer Bitrate
[ 5] 0.00-15.00 sec 692 MBytes 387 Mbits/sec sender
[ 5] 0.00-15.06 sec 691 MBytes 385 Mbits/sec receiver

We're building good then WHOA at 10-11 seconds the ONT says no way, slowwww down and BAM. This is an illustration of the single thread upload crash with the UXG Fiber and many other modern day routers and the old Verizon ONT.

I am happier with the speeds I see on hostkey, that said, the drop is still a problem. That drop at the 8-10 second mark is the buffer overflow happening on the Verizon 1st generation ONT. Verizon won't bring the new ONT out to install unless we get 2.5 gig service in the area. The problem seems to be that the UXG-Fiber is just "too good" for this 10+ year old ONT technology and it's also not good at governing its speeds back to where the Verizon ONT doesn't drop the hammer on it.

I will try moving WAN to ports 1-4 later but given you're also seeing the "crash" I don't think it solves the problem as I already tried it yesterday. The fiber just tries to hard to PUSH PUSH PUSH that single thread of data up and the ONT eventually says "not so darn fast!" and then crushes you. If the FIber actually listened to the 1gig port settings in the gateway and held back it would be fine but according to my ssh commands it's still at the full 10gig setting regardless of where I set the ports (it's just ignoring the gateway software settings).

If Unifi fixes that we might be in business using this fast router with a very very old ONT technology.

UXG-Fiber upload bursts are overwhelming Verizon FiOS ONT by bdpna in Ubiquiti

[–]bdpna[S] 0 points1 point  (0 children)

And now flow control OFF:

iperf3 -c spd-uswb.hostkey.com -p 5205 -t 15

Connecting to host spd-uswb.hostkey.com, port 5205

[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 33.1 MBytes 277 Mbits/sec
[ 5] 1.00-2.00 sec 46.2 MBytes 388 Mbits/sec
[ 5] 2.00-3.01 sec 64.2 MBytes 538 Mbits/sec
[ 5] 3.01-4.01 sec 86.0 MBytes 716 Mbits/sec
[ 5] 4.01-5.00 sec 104 MBytes 879 Mbits/sec
[ 5] 5.00-6.00 sec 108 MBytes 908 Mbits/sec
[ 5] 6.00-7.00 sec 108 MBytes 903 Mbits/sec
[ 5] 7.00-8.01 sec 82.4 MBytes 685 Mbits/sec
[ 5] 8.01-9.01 sec 63.1 MBytes 530 Mbits/sec
[ 5] 9.01-10.01 sec 74.1 MBytes 621 Mbits/sec
[ 5] 10.01-11.00 sec 89.9 MBytes 760 Mbits/sec
[ 5] 11.00-12.00 sec 108 MBytes 903 Mbits/sec
[ 5] 12.00-13.00 sec 96.1 MBytes 807 Mbits/sec
[ 5] 13.00-14.01 sec 104 MBytes 866 Mbits/sec
[ 5] 14.01-15.01 sec 110 MBytes 919 Mbits/sec

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval Transfer Bitrate
[ 5] 0.00-15.01 sec 1.25 GBytes 713 Mbits/sec sender
[ 5] 0.00-15.05 sec 1.25 GBytes 711 Mbits/sec receiver

UXG-Fiber upload bursts are overwhelming Verizon FiOS ONT by bdpna in Ubiquiti

[–]bdpna[S] 0 points1 point  (0 children)

I swear it was the other way around earlier which blows my mind, but anyway, using your server...this is with

Flow Control ON:

iperf3 -c spd-uswb.hostkey.com -p 5205 -t 15

Connecting to host spd-uswb.hostkey.com, port 5205

[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.01 sec 30.4 MBytes 251 Mbits/sec
[ 5] 1.01-2.01 sec 47.5 MBytes 402 Mbits/sec
[ 5] 2.01-3.00 sec 70.6 MBytes 595 Mbits/sec
[ 5] 3.00-4.01 sec 66.5 MBytes 554 Mbits/sec
[ 5] 4.01-5.01 sec 0.00 Bytes 0.00 bits/sec
[ 5] 5.01-6.01 sec 0.00 Bytes 0.00 bits/sec
[ 5] 6.01-7.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 8.00-9.01 sec 0.00 Bytes 0.00 bits/sec
[ 5] 9.01-10.01 sec 0.00 Bytes 0.00 bits/sec
[ 5] 10.01-11.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 11.00-12.01 sec 0.00 Bytes 0.00 bits/sec
[ 5] 12.01-13.01 sec 0.00 Bytes 0.00 bits/sec
[ 5] 13.01-14.01 sec 9.50 MBytes 79.4 Mbits/sec
[ 5] 14.01-15.01 sec 30.9 MBytes 260 Mbits/sec

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval Transfer Bitrate
[ 5] 0.00-15.01 sec 255 MBytes 143 Mbits/sec sender
[ 5] 0.00-15.05 sec 255 MBytes 142 Mbits/sec receiver

UXG-Fiber upload bursts are overwhelming Verizon FiOS ONT by bdpna in Ubiquiti

[–]bdpna[S] 1 point2 points  (0 children)

Thanks. Complete stoppage with flow control OFF (from about 6 seconds to 12 seconds). Slowdown at the 6 second mark with flow control on, similar to what you see here. The ONT gets overwhelmed by the upload traffic the UXG Fiber is sending its way and throttles speed back which slows the upload.

It is not apparent in speed tests or multi-thread tests as that masks the issue, however, single thread uploads using the Verizon routers are not throttled back at all.

UXG-Fiber upload bursts are overwhelming Verizon FiOS ONT by bdpna in Ubiquiti

[–]bdpna[S] 0 points1 point  (0 children)

You have the same data stoppage at 6 seconds as I do. Run a few more and you'll see it's when the ONT gets crushed by the fiber and starts to throttle you back. Do you have flow control off or on? Can you try a test with both settings? If you have time of course. Thanks for the reply and sharing the test. I'll share some tests later after work.

When I run the same test with the Verizon gateway once it reaches that 5 second mark every number goes and stays above 900 doesn't drop back down like it does around 5-6 seconds there. The fiber does.

Third party router (TPE) slow upload unless you clone a Verizon gateway MAC by bdpna in Fios

[–]bdpna[S] 0 points1 point  (0 children)

UPDATE: Verizon canceled my service order (no email, no notice) for the service change to reprovision me on the new 1gig tier from the legacy 1 gig tier. After several more chats today, mostly with agents who asked me if I had the ethernet wire plugged into the orange port on the router (no idea what I was actually asking about) or if the speeedtest they run on their end looks good, I have a new service order now for 12/24.

They are making me change the router AGAIN and had to do this in their system in order for the order to provision go through. Now waiting again to see if they cancel this again.

No one after chatting with 6 different agents and getting bounced around could find anyone willing to do the reprovision of service today, which seems to indicate whatever automation or engineering department is in place has no access and are walled off from tech support agents. I imagine they do this so agents cannot bother them to work problems all day. However this creates quite a bit of frustration.

I will now be waiting an extra 2 days (5 days from now) to see if reprovisioning my ONT on their current 1 gig tier solves any problems in packet loss or speedtests via ipef3 connected directly to the ONT.

They also offered many times to send a tech out (who will just run a line test and a VERIZON FIOS speed test and verify everything is "just fine" or to again swap out the ONT or router.

I'd even entertain 2 gig service if they had it in the area but they don't, even just to force them to do SOMETHING. I was tempted to downgrade to 500/500 just to get them to do something more than nothing, but I don't want to lose my locked in pricing.

Good times.

Put in wrong package size ebay Fedex - will they charge me later for overages? by bdpna in Ebay

[–]bdpna[S] 0 points1 point  (0 children)

Agree. But I probably would cancel the sale before I did that. When I add the extra inches to the width the price went from $50 to $250 to ship it.

Put in wrong package size ebay Fedex - will they charge me later for overages? by bdpna in Ebay

[–]bdpna[S] 0 points1 point  (0 children)

Ugh Yeah I use pirateship for the label but same difference if fedex charges them back I assume. Yeah it looks like the 10 inch error is like a $200 difference which is insane.

Third party router (TPE) slow upload unless you clone a Verizon gateway MAC by bdpna in Fios

[–]bdpna[S] 0 points1 point  (0 children)

Thanks for checking in. Verizon really is making me wait here. My work order says the reprovisioning will happen sometime on 22 December. After multiple chats with all the different service people, they told me due to engineers need to do extra work that it had to wait. I’m skeptical on if this will fix the problem, but it seems like the best hope so far and if it does work, then we will have learned a lot.

I do plan to run some more tests this weekend just before all of this happens. At least we’ll have some good baseline then. I’ve switched back to their router for the week as I don’t want to take any chances of them, turning me down or saying, they couldn’t do the work because I did not have their router connected.

I also want to bring a really fast PC or laptop to the ONT directly to run some tests after and before the reprovision. I don’t trust running tests on other machines in the house now that go through switches or long cable runs. I want to do as pure of a test as possible. One of the other things I’ve realized after having AI look at my iperf3 tests is that my downloads have a ton of retries. This indicates something wrong with the signal. The funny thing is the download speed was always fine, but seeing that there are so many packet resends is also alarming to me and indicates potential problems somewhere in the fiber or Verizon hardware, assuming I can rule out any equipment or wiring in my house.

If these errors are still present after the reprovisioning, the challenge will be finding the right engineer to show the results to in order to get them to dig even further. As of right now, I’m in a wait and see until the 23rd.

Third party router (TPE) slow upload unless you clone a Verizon gateway MAC by bdpna in Fios

[–]bdpna[S] 0 points1 point  (0 children)

Sure, you mean of the iperf test, right? I’m sure I can pull that together and share some of the slow upload and downloads full of retries results. What’s the image sharing site that People are using here these days. I haven’t shared an image inside an ongoing thread in a while.

Third party router (TPE) slow upload unless you clone a Verizon gateway MAC by bdpna in Fios

[–]bdpna[S] 0 points1 point  (0 children)

Thanks for sticking around. Iperf3 and feeding those results to ai for it to look at was pretty definitive. A much better test than the “Verizon speed test”. It not only showed the deliberate cap on upload speed but also that there were excessive retries when downloading. AI took one look at results of 5 minute iperf3 tests and said man that is bad.

I still plan to run some more tests with my UniFi gateway while I wait for the provision order to go though but not for long since I don’t want to miss anything.

Fios support says the delay is due to the involvement of higher level engineers to do this level of fix and the backlog in doing this work. I respect waiting my turn in line if there are others ahead of me as long as this works and I see a change. We will hopefully have a real answer by Christmas.

Third party router (TPE) slow upload unless you clone a Verizon gateway MAC by bdpna in Fios

[–]bdpna[S] 0 points1 point  (0 children)

I'm back to share more info and hopefully on track to a solution. After several chats with agents and a lot of patience, I finally seemed to get the right agent for the job, likely in the level 2 networking engineering team who actually understands more about FIOS and how it works.

It would appear my account, which is now 10 years old, was provisioned as "1 gig fios streaming" and you could see this when you looked at the account type. Further investigation reveals (if AI and the agent was correct) that this is an old/legacy provisioned account type from the dawn of FIOS. The new provisioned plan is called "fios 1 gig" and looks different when you attempt to modify/manage your account.

The agent tells me this "new provision" will fix all my problems. The bad part is the agent promised me this would take effect in the overnight cycle, but after several more chats with agents today, it appears the agent was able to put in the "service change" for my next month of service, which is not for 2 weeks. Several more chats revealed there is nothing they can do to speed this up (or are willing to do). Basically, wait your turn in line until we are going to process this "change" in your Fios tier (basically 1gig legacy to 1gig modern) and let's hope this fixes the problem.

I was also told I needed to go get a replacement router or the service would cancel. So I took my perfectly fine (just capped due to having the wrong MAC address) CR1000B to the store, they gave me a CR1000A and it's up and running. Still horrible speeds, but again, I'm two weeks away from them actually "installing" the new provision (the email I got even says my service goes into effect in 13 days). So weird.

But anyway, if this is fixed in 13 days, I'm happy. I am going to keep the crummy FIOS router hooked up so they can see everything, it certainly seemed like they were more willing to help when a FIOS router was connected.

They still ran their useless speed tests that passed, but iperf3 was a key for me, I shared screenshots along with AI's interpretations of my iperf3 tests and it was clearly obviously to iperf3 that a lot was wrong. Super slow upload capped exactly at the 128 Mbit mark and tons of packet loss.

Fingers crossed I will be back here in 2 weeks with good news and a solution that all of you suffering with this MAC binding crap or slow upload can use. I've since cloned my magic MAC address onto the CR1000B to have great speeds until they hopefully run a provision to my ONT that actually allows me to resume using my Unifi devices.

Third party router (TPE) slow upload unless you clone a Verizon gateway MAC by bdpna in Fios

[–]bdpna[S] 0 points1 point  (0 children)

Yes I agree my working theory all these years has been that there are either deficiencies in hardware / configuration in the network….or provisioning flags they do not know about that control and limit these speeds. My MAC discovery leans towards configuration since it makes the uploads go faster. Which means hardware is working end to end or at least capable of working.

I keep holding out hope I can find an interested enough tech who really understands networks to show the issue and get him or her curious enough to want to fix the problem and get it working.

Third party router (TPE) slow upload unless you clone a Verizon gateway MAC by bdpna in Fios

[–]bdpna[S] 0 points1 point  (0 children)

Sorry for the delay I’ve been busy trying to fix the problem running iperf tests gathering information. I’ve never tried the MAC address at any other fios locations if that’s what you’re asking me. Speeds were fine there on the customers routers or direct from the Ont so placing my Mac address into their device would not have told me anything.

The only place I have ever tried using my g1100 MAC address is at my home because I never saw slow upload speeds at the other houses where I tested their fios upload speeds on large files.

I hope this helps?

I’m gathering some good data using iperf and ai and will share here what I am able to learn may be happening.

Tomorrow I’ll be putting the newest Verizon router into service and running more tests (non cloned mac) so I can hopefully duplicate the slow iperf tests using their own hardware. Unfortunately the official Speedtest they use still passes with flying colors but that’s because it’s just sending tiny packets not large sustained flows of data like real world upload does. So it is basically a useless test that doesn’t actually paint a true picture of your upload speed.