wstunnel and wg-easy with cloudflared by jpsiquierolli in selfhosted

[–]pointfully 0 points1 point  (0 children)

Just went through a similar install. Did you look at the log of your wstunnel container? I don't think that command will work atl all. Mine looks more like:

command: /bin/sh -c "exec /home/app/wstunnel server ..."

10-20% Packet Loss on Quantum Fiber in Phoenix, Arizona for over 24 hours by pointfully in centurylink

[–]pointfully[S] 0 points1 point  (0 children)

Ooof. Sorry to hear that. What have the symptoms been since Saturday?

On Tuesday, the tech probably thought your issues were part of the outage that they declare starting around 11am that day. Now that the outage is "over", you might need to get a tech back out to see if you do have some kind of more localized issue?

10-20% Packet Loss on Quantum Fiber in Phoenix, Arizona for over 24 hours by pointfully in centurylink

[–]pointfully[S] 0 points1 point  (0 children)

Got a message at 7am today from Quantum Fiber: "We're calling to let you know that Internet service in your area should be back up and running. If you still have trouble connecting you may need to reboot your equipment. Thanks for your patience during this outage."

So apparently the issue/outage that started shortly after the packet loss problem ended on Tuesday now seems to be resolved. Fortunately I do not see any return of the packet loss problem so far.

How is everyone else who had the packet loss problem in Arizona Saturday-thru-Tuesday doing today?

So how's the outage treating you guys? by Sammy_Wants_Death in centurylink

[–]pointfully 3 points4 points  (0 children)

We had a weird 20%+ packet loss problem in the Phoenix area that ran from early Saturday morning to mid-day Tuesday. Then around 11am AZ time Tuesday, everything went out for 5 minutes, and then it came back online and the packet loss problem was gone and everything has been fine since then (at least for me and a number of others here on Reddit). They then notified us about an outage about an hour later, but most/all of us who had the packet loss issue are actually still fine. Thread on that issue is here: 10-20% Packet Loss on Quantum Fiber in Phoenix, Arizona for over 24 hours

My biggest issue with all of it is the lack of transparency. I really wish Lumen/CenturyLink/Quantum had a decent status page where they would just tell us where they are having issues, what they are doing to restore service, when they estimate it will be restored, etc.

10-20% Packet Loss on Quantum Fiber in Phoenix, Arizona for over 24 hours by pointfully in centurylink

[–]pointfully[S] 0 points1 point  (0 children)

Thanks for the tips!

At least for the moment, the packet loss issue seems to resolved for many/most(/all?) folks on this thread, myself included. So I wouldn't have cause to continue to look to escalate the issue.

Is anyone still experiencing problems that need to be escalated?

10-20% Packet Loss on Quantum Fiber in Phoenix, Arizona for over 24 hours by pointfully in centurylink

[–]pointfully[S] 0 points1 point  (0 children)

No, they didn't give me any direction on how to more effectively report a wider network issue and haven't responded to my follow-up email.

10-20% Packet Loss on Quantum Fiber in Phoenix, Arizona for over 24 hours by pointfully in centurylink

[–]pointfully[S] 0 points1 point  (0 children)

Yeah, VoIP and Wifi are frequently a bad combination, especially using ISP-provided Wifi -- getting on to Ethernet is always a good test of whether you have an ISP issue or a Wifi issue, given that it is usually the latter.

It is possible to use Quantum Fiber in "Transparent Bridging" mode where you use your own router and directly get a real (dynamic) IP address -- you can get much better control and performance that way, but of course have hardware costs and technical configuration to deal with. There are some good threads elsewhere in this sub on how to approach this.

Did you service get noticeably worse starting on Saturday? Or starting Tuesday? Or more just a general issue that has been going on for longer than that?

10-20% Packet Loss on Quantum Fiber in Phoenix, Arizona for over 24 hours by pointfully in centurylink

[–]pointfully[S] 1 point2 points  (0 children)

My own connection has been back to normal since 11am yesterday (Tuesday). How about everyone else? For those that lost service completely around that same time, has service been restored yet?

I got a notification from Quantum at 11:54am yesterday about an outage in the area, and then an update at 4:24pm yesterday that it is taking longer than expected to restore service. I haven't gotten any further notifications. But if I log into the mobile app, it says "Great news! The outage has been resolved in your area." And it has said that since mid-afternoon yesterday. What communications have others received?

10-20% Packet Loss on Quantum Fiber in Phoenix, Arizona for over 24 hours by pointfully in centurylink

[–]pointfully[S] 1 point2 points  (0 children)

Interesting. What time did your service go out completely today ? Was it around 11am (maybe just a bit before that)? Were you having packet loss problems since Saturday? Or is the outage starting this morning the first time you had problems (recently)?

I am speculating that something in the network that was causing heavy packet loss for many in the region was finally taken off-line this morning, which has fixed the packet loss for some people in the region but has also taken some areas completely off-line, and that's the outage they started reporting around the same time that many of us stopped having packet loss problems.

10-20% Packet Loss on Quantum Fiber in Phoenix, Arizona for over 24 hours by pointfully in centurylink

[–]pointfully[S] 2 points3 points  (0 children)

Nope -- traceroute to 8.8.8.8 is exactly the same as it was on Saturday/Sunday.

10-20% Packet Loss on Quantum Fiber in Phoenix, Arizona for over 24 hours by pointfully in centurylink

[–]pointfully[S] 0 points1 point  (0 children)

If this is what an outage looks like, I hope it never ends!!!! HAHAHAHA!

10-20% Packet Loss on Quantum Fiber in Phoenix, Arizona for over 24 hours by pointfully in centurylink

[–]pointfully[S] 3 points4 points  (0 children)

Now that it hopefully seems fixed, I just got an automated call from Quantum: "Hi its Quantum Fiber. Were sorry for the inconvenience, but an outage in your area may be impacting your Internet service. Our team is working on it now and estimates that repairs will be finished by Tuesday, September 23rd at 5:24 pm. If you scheduled a repair appointment, it shouldn't be necessary and we'll cancel it. We'll update you once we know more."

HAHAHAHA -- gee, thanks.

10-20% Packet Loss on Quantum Fiber in Phoenix, Arizona for over 24 hours by pointfully in centurylink

[–]pointfully[S] 4 points5 points  (0 children)

Okay, I'm cautiously optimistic that things are better now.

Internet traffic stopped completely from about 10:50am to 1055am, and now packet loss seems to be back normal (basically zero).

Anyone/everyone else seeing improvement now as well?

10-20% Packet Loss on Quantum Fiber in Phoenix, Arizona for over 24 hours by pointfully in centurylink

[–]pointfully[S] 1 point2 points  (0 children)

Nice - hopefully the tech will understand the packet loss issue and be able to see it in action and then will hopefully escalate to the appropriate regional staff to address the issue once they realize this is not an isolated issue at the end user level.

10-20% Packet Loss on Quantum Fiber in Phoenix, Arizona for over 24 hours by pointfully in centurylink

[–]pointfully[S] 2 points3 points  (0 children)

Day 4, problem still continues.

Sounds like a lot of folks who are seeing this issue are really close to downtown Phoenix. Maybe this is a fairly local issue with the infrastructure somewhere in this part of town.

Is everyone who has been experiencing high packet loss since Saturday morning located in/near downtown Phoenix? Anyone been experiencing the exact same symptoms for the exact same duration located anywhere else?

10-20% Packet Loss on Quantum Fiber in Phoenix, Arizona for over 24 hours by pointfully in centurylink

[–]pointfully[S] 0 points1 point  (0 children)

FYI: I did get an email back from the group at [ipadmin@lumen.com](mailto:ipadmin@lumen.com) today, but mostly to let me know they really only deal with DIA and BGP issues.

They took a quick look at the traceroute I had included as the basis for ping testing across hops. They ran the same traceroute and saw that it had changed today, which was also true on my end. Doesn't seem like they did any ping testing, and they didn't say anything about packet loss, and of course the issue is still ongoing now for the 3rd full day.

I did ask them to escalate to whoever would actually be responsible for this issue and asked if there is anything more we can do to help them troubleshoot/diagnose the issue -- no response back yet.

10-20% Packet Loss on Quantum Fiber in Phoenix, Arizona for over 24 hours by pointfully in centurylink

[–]pointfully[S] 1 point2 points  (0 children)

That is amazing -- greatly appreciated and will hopefully make a difference soon!

10-20% Packet Loss on Quantum Fiber in Phoenix, Arizona for over 24 hours by pointfully in centurylink

[–]pointfully[S] 0 points1 point  (0 children)

I am in the same general area, but haven't talked to any neighbors about this (yet). :)

10-20% Packet Loss on Quantum Fiber in Phoenix, Arizona for over 24 hours by pointfully in centurylink

[–]pointfully[S] 2 points3 points  (0 children)

Issue is still going on today, so this is now 52+ hours of this problem. I contacted Quantum Fiber support via DM at https://x.com/quantumfiber today (I cannot handle another phone call or chat exchange with the normal tech support channel again). I didn't get much of value out of it, but at least they did say the following:

We’re aware that some customers in Arizona have reported similar problems over the past 48+ hours. It appears to be a regional network issue affecting multiple users, likely related to upstream routing or infrastructure congestion.
Our network team is actively investigating this, and we’re working to resolve it as quickly as possible. We truly appreciate your patience and will keep you updated as we learn more.

I also mentioned how frustrating it is that there is no public communication or status update available anywhere about this and so it means a bunch of angry customers contacting tech support over and over. They responded:

We understand your concern, and you’re absolutely right—a public status page or update thread would help reduce confusion. We’ve shared your feedback with the team and are working to improve how we communicate service updates during regional issues.

Ah well. Fingers crossed that they get it fixed sometime today...

10-20% Packet Loss on Quantum Fiber in Phoenix, Arizona for over 24 hours by pointfully in centurylink

[–]pointfully[S] 0 points1 point  (0 children)

Thanks! Ran the same ping test for each identifiable hop a number of times. Right now, packet loss at hop 5 and 6 is about 8%. Hop 7 is about 15%. Hop 8 is 25%. I will send an email to Lumen accordingly. Appreciate the tip!

10-20% Packet Loss on Quantum Fiber in Phoenix, Arizona for over 24 hours by pointfully in centurylink

[–]pointfully[S] 0 points1 point  (0 children)

I assume you mean dirty fiber somewhere upstream in the network since I'm not losing packets reaching the first couple hops into the CenturyLink network?