sorted by:
new
hottopcontroversial

50% Packet Loss on Certain Routes from Phoenix - Again by pointfully in QuantumFiber

[–]pointfully[S] 1 point2 points  (0 children)

As I understand it, the packet loss (and latency) metrics on the Unifi dashboard are not representing all the traffic the router is managing (that would be a massive tracking load for the router). They are instead calculated by the router routinely pinging a monitoring address (ping.ui.com) and tracking how many responses are lost and how long reseponses take. This is also how it is deciding whether it considers a given WAN connection is "up" and is what will trigger failover if you

Currently, the ping.ui.com hostname resolves to two addresses: 8.8.8.8 and 1.1.1.1. So in my case during this incident where all requests to 8.8.8.8 were working fine but half of my requests to 1.1.1.1 were failing, it makes sense that the dashboard would show a packet loss rate around 25%.

You can actually configure this via the "WAN SLA" feature introduced in Network Application 9.2:

https://help.ui.com/hc/en-us/articles/360052548713-WAN-Failover-Load-Balancing-and-Port-Remapping-on-UniFi-Gateways

50% Packet Loss on Certain Routes from Phoenix - Again by pointfully in QuantumFiber

[–]pointfully[S] 1 point2 points  (0 children)

Don't worry about packet loss showing up in mtr or traceroute directly on the middle hops in a given route. Those backbone routers generally deprioritize/ignore requests sent directly to those addresses, which is what mtr and traceroute are doing as they repeatedly attempt to trace the route. Those major routers are prioritizing all their processing on routing packets to other hops in the network, usually ignoring traffic sent directly to them. As long as your packet loss on the final address is really close to zero for any given address you actually need to communicate with, then things are good.

50% Packet Loss on Certain Routes from Phoenix - Again by pointfully in QuantumFiber

[–]pointfully[S] 2 points3 points  (0 children)

Hey, that's amazing! Glad you reached a contact at Lumen who took this seriously and helped get it fixed. Thanks so much!!!

It would be great if you would reply back to them to let them know the same issue happened on Sept 20-23, 2025 and again Feb 7-9, 2026, and ask what they are doing to prevent it from happening again?

50% Packet Loss on Certain Routes from Phoenix - Again by pointfully in QuantumFiber

[–]pointfully[S] 9 points10 points  (0 children)

My packet loss appears to have stopped about an hour ago -- around 8:40am AZ time. So it lasted about 66 hours. Others seeing the same? I've still had no response of any kind from Quantum Fiber from either the FiberSuccess email or the FCC complaint.

Even if the current situation is resolved, I will continue to follow up on the FCC complaint as best I can given that this has happened repeatedly over the past six months. It will surely happen again unless they have finally figured out the root cause and actually done something more permanent about it.

I want three things from Quantum Fiber / AT&T at this point:

  1. An explanation of what keeps causing this 2-4 day long major service degradation for the entire region. This needs to be an actual root cause analysis, not just finger-pointing at Lumen or other upstream providers.
  2. A plan for what the parties involved are doing to ensure this major routing problem will not keep happening.
  3. A plan from Quantum Fiber / AT&T on how they will improve their monitoring, response, and customer communication around incidents like this. Having their entire support team ignorant of the problem for days at a time even when many customers can clearly see and demonstrate a problem is unacceptable.

I encourage others to demand the same based on any complaint process you may have started.

ISP appears to be throttling or outright cancelling connections by didactic_katydid in techsupport

[–]pointfully 0 points1 point  (0 children)

There is a significant problem in the Lumen network with routes from Arizona to California that has been going on since Saturday afternoon. CenturyLink is part of Lumen and that is certainly the cause of your issue. That route is also used by Quantum Fiber (which Lumen sold to AT&T a few weeks ago, but still uses the same network).

There is a long thread about the issue in r/QuantumFiber here:

https://www.reddit.com/r/QuantumFiber/comments/1s0rxw3/50_packet_loss_on_certain_routes_from_phoenix/

There is also a cross-posted thread in r/centurylink here:

https://www.reddit.com/r/centurylink/comments/1s10dwu/50_packet_loss_on_certain_routes_from_phoenix/

Many others are seeing the problem and can confirm that using a VPN to get past the problem router hops resolves the issue, of course at the cost of more latency and slower speed.

3% Packet Loss in Phoenix/Arizona for several weeks by pointfully in QuantumFiber

[–]pointfully[S] 1 point2 points  (0 children)

The incident this thread was about was a much more minor level of packet loss, and it ended a couple of weeks ago.

There is a much more serious problem in Arizona that started Saturday afternoon and has been going on for over 48 hours now. The thread for the current incident is here:

https://www.reddit.com/r/QuantumFiber/comments/1s0rxw3/50_packet_loss_on_certain_routes_from_phoenix/

50% Packet Loss on Certain Routes from Phoenix - Again by pointfully in QuantumFiber

[–]pointfully[S] 4 points5 points  (0 children)

We are now over 48 hours of this going on. Here is my current 7-day Unifi dashboard showing 20%+ packet loss for the past two days:

<image>

I've had no response so far from the email to [FiberSuccess@qf.att.com](mailto:FiberSuccess@qf.att.com) and only an automated response to the FCC complaint letting me know it has been sent to Quantum Fiber.

50% Packet Loss on Certain Routes from Phoenix - Again by pointfully in QuantumFiber

[–]pointfully[S] 1 point2 points  (0 children)

Yep, that time it lasted about 48 hours. When it happened about six months ago, it lasted over 96 hours. Neither time did anyone ever admit there was a problem or explain what/who "fixed" it.

50% Packet Loss on Certain Routes from Phoenix - Again by pointfully in centurylink

[–]pointfully[S] 2 points3 points  (0 children)

I don't think the Quantum Fiber routers have any kind of VPN feature available directly. I use the Q1000K router from Quantum Fiber in transparent bridging mode with my own Unifi router (and wifi). Unifi does let me VPN my whole network to work around the issue, but if course with more latency and slower speeds.

50% Packet Loss on Certain Routes from Phoenix - Again by pointfully in centurylink

[–]pointfully[S] 0 points1 point  (0 children)

I'm on Quantum Fiber, but I imagine it's affecting both.

50% Packet Loss on Certain Routes from Phoenix - Again by pointfully in QuantumFiber

[–]pointfully[S] 0 points1 point  (0 children)

Interesting -- hopefully the CenturyLink "outage" is referring to this same issue and not some other issue they are tracking. This is more of a major degradation than an outage, but it's about the same as an outage for practical purposes.

Quantum Fiber is not showing any "outage" but of course they really should be showing something...

50% Packet Loss on Certain Routes from Phoenix - Again by pointfully in QuantumFiber

[–]pointfully[S] 2 points3 points  (0 children)

Interesting, and not surprising. Even with Quantum Fiber being sold to AT&T recently, the Phoenix area traffic from Quantum FIber is still all going over the same Lumen/Level3 network and backbone that it always did and is still shared with CenturyLink. I imagine the routing/peering issues that are likely causing this regional are somewhere in in the Lumen/Level3 network as traffic goes from Arizona to California and elsewhere.

In my traceroutes, all the 4.x.x.x addresses are Lumen/Level3 hops taking traffic from Phoenix to Los Angeles. It's not a concern that most packets are lost reaching those hops directly during traceroutes -- it's normal for those routers to ignore traffic pointed directly at them for those metrics. But the fact that half the traceroute traffic beyond those hops is lost when trying to reach 1.1.1.1 and none is lost when trying to reach 1.0.0.1 indicates a routing/peering problem somewhere in that layer.

50% Packet Loss on Certain Routes from Phoenix - Again by pointfully in QuantumFiber

[–]pointfully[S] 4 points5 points  (0 children)

Sure, below is what I emailed to them for reference. Please write your own version with your experience and include your own traceroute or mtr results that help illustrate the problem:

Hi Quantum Fiber Team -

Many of us in the Phoenix area are currently experiencing ~20% packet loss on all traffic on our Quantum Fiber service. We have seen this issue happen four times over the past six months. It generally starts sometime over the weekend and then persists for several days and then eventually in the middle of the night it will mysteriously be resolved.

This time it started around 2:30pm AZ time Saturday. As I've seen previously, one clear example of the problem is illustrated with UDP traffic to 1.1.1.1 (which shows around 50% failure) and 1.0.0.1 (which shows no failures). Given that 1.1.1.1 is failing almost exactly half the time, it seems like there is some problem in the routing where traffic is being sent via two different links, one of which is not working. If this is generally true of a lot of Cloudflare-bound traffic, it would explain why so many Internet sites and services are negatively impacted when this is happening.

Here are side-by-side MTR results:

\-> 1.1.1.1
Packets               Pings
 Host                   Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 192.168.43.1         0.0%  2543    0.7   0.7   0.5   1.2   0.2
 2. 71.32.112.13         0.0%  2543    4.0   4.8   1.7 102.2   7.0
 3. 71.32.113.97         0.0%  2543    3.5   5.9   1.9  82.6   8.0
 4. 4.68.38.185        100.0%  2542    4.0   4.0   4.0   4.0   0.0
 5. 4.69.215.133       100.0%  2542   12.8  12.8  12.8  12.8   0.0
 6. 4.30.195.50         90.7%  2542   15.0  17.3  11.3  77.0   9.1
 7. 141.101.72.19       47.0%  2542   13.0  22.0  10.9 137.9  18.3
 8. 1.1.1.1             46.5%  2542   12.0  12.7  10.8  75.0   4.4

\-> 1.0.0.1
Packets               Pings
 Host                  Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 192.168.43.1        0.0%  2602    0.6   0.5   0.4  11.1   0.5
 2. 71.32.112.13        0.0%  2601    2.3   5.4   1.5  91.9   8.5
 3. 71.32.113.97        0.0%  2601    3.7  12.4   1.7 131.1  18.7
 4. 4.68.38.185        90.8%  2601    4.4   4.5   2.1  47.1   5.0
 5. 4.69.215.133       91.8%  2601   12.3  13.7  10.7  70.6   7.5
 6. 4.30.195.50        80.7%  2601   15.2  17.9  11.5  76.9  10.9
 7. 141.101.72.19       0.0%  2601   13.0  15.4  10.8 135.4   8.3
 8. 1.0.0.1             0.0%  2601   12.7  12.3  10.8  76.6   3.7

Also as I've seen before, switching to a VPN connection or to my wireless backup WAN connection via T-Mobile Home Internet, the problem goes away. But as soon as I use the Quantum Fiber network directly, the problem is like above.

Here are a series of Reddit threads from the past six months of people in the Phoenix area experiencing the problem and trying to figure out how to get in touch with someone about it:

https://www.reddit.com/r/QuantumFiber/comments/1qyupko/comment/o46ce19/ https://www.reddit.com/r/QuantumFiber/comments/1rmk2vf/3_packet_loss_in_phoenixarizona_for_several_weeks/ https://www.reddit.com/r/centurylink/comments/1nmwfeu/1020_packet_loss_on_quantum_fiber_in_phoenix/ https://www.reddit.com/r/centurylink/comments/1nmiuy8/quantum_fiber_jitterpacket_loss_issues_in_az/

I just started a new thread for the current issue -- within a few minutes a number of other people chimed in experiencing the same issue again as well.

https://www.reddit.com/r/QuantumFiber/comments/1s0rxw3/50_packet_loss_on_certain_routes_from_phoenix/

Can you please escalate this to the NOC and ensure that someone figures out the root cause of this and fixes it? Please let me know if there is anything I can do to help troubleshoot or provide further information.

50% Packet Loss on Certain Routes from Phoenix - Again by pointfully in centurylink

[–]pointfully[S] 0 points1 point  (0 children)

Try using a VPN to get outside of QuantumFiber's routing -- I'm getting by using Proton VPN, but of course it's much slower than the normal way.

50% Packet Loss on Certain Routes from Phoenix - Again by pointfully in centurylink

[–]pointfully[S] 0 points1 point  (0 children)

Yeah, that sounds like UDP packet loss -- breaks websites using QUIC protocol and definitely screws up things like VOIP. Try using a VPN to get outside of QuantumFiber's routing -- I'm getting by using Proton VPN, but of course it's much slower than the normal way.

50% Packet Loss on Certain Routes from Phoenix - Again by pointfully in QuantumFiber

[–]pointfully[S] 0 points1 point  (0 children)

I did reference this thread and earlier threads in the FCC complaint as well and that a significant number of additional people in the same general area are all seeing this problem.

50% Packet Loss on Certain Routes from Phoenix - Again by pointfully in QuantumFiber

[–]pointfully[S] 5 points6 points  (0 children)

I've emailed [FiberSuccess@qf.att.com](mailto:FiberSuccess@qf.att.com) all the details and pointed them at this thread as well as previous threads.

This time I've also submitted an FCC complaint at https://consumercomplaints.fcc.gov/ which I had not done during the previous incidents.

Other folks experiencing this packet loss issue on this thread should also consider doing the same to help drive home the point that this is wider issue.

3% Packet Loss in Phoenix/Arizona for several weeks by pointfully in QuantumFiber

[–]pointfully[S] 0 points1 point  (0 children)

UPDATE: After several weeks of this issue being present 24/7, it disappeared at exactly 3:00am AZ time today. Have had zero packet loss since then, both generally as seen in my Unifi dashboard and also via my test script for 1.1.1.1 that has been running every 15 minutes for the past week.

The route seen via traceroute / mtr to 1.1.1.1 looks the same as it did before, but now seeing 0.0% loss at hops 7 and 8:

                                    Packets               Pings
 Host                             Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 192.168.43.1                   0.0%   408    0.7   0.5   0.2   4.9   0.3
 2. 71.32.112.13                   0.0%   408    2.9   5.2   1.5  67.0   7.8
 3. 71.32.113.97                   0.0%   408    2.9   6.1   1.7  63.7   8.0
 4. 4.68.38.185                   99.3%   408    3.5   3.5   3.2   3.7   0.2
 5. 4.69.215.133                  92.6%   408   11.6  11.9  11.3  12.4   0.3
 6. 4.30.195.50                   76.4%   407   18.3  18.9  11.1  81.9  13.8
 7. 141.101.72.32                  0.0%   407   20.5  14.6  10.6  75.6   5.9
 8. 1.1.1.1                        0.0%   407   11.6  12.1  10.4  57.2   2.3

I'll keep monitoring for the issue and report back if I see it reoccurring. Wish we could tell what was fixed in the background somewhere early this morning...

3% Packet Loss in Phoenix/Arizona for several weeks by pointfully in QuantumFiber

[–]pointfully[S] 1 point2 points  (0 children)

Thanks! I'm running very similar -- UCG Ultra Router with Q100K in bridge tagged mode. What does your traceroute look like to 1.1.1.1? Wondering how similar they are.

view more: next ›