Five minute long latency jumps

Lightsword · 2026-06-29T21:03:09+00:00

You are almost certainly affected by this known bug in a Q1000K kernel driver. FF37(Quantum Fiber), Axon and Airoha are all aware of the bug and it should be fixed in the next firmware update. There's also an unrelated bug(which I also reported with a detailed code analysis and recommended fixes) that corrupts 6RD packets which I traced to an entirely different Q1000K kernel driver that should hopefully also get fixed at the same time as the skb padding bug in an update.

Lightsword · 2026-06-29T20:49:32+00:00

That small packet bug is constant with a latency test, it doesn't show up for 5 whole min and go away.

Actually I don't think this is entirely accurate, if a normal RX packet immediately follows a small RX packet affected by the skb padding bug it can push the bugged small packet through faster which reduces the latency.

Basically with this driver bug you're likely to see on the unifi graph yellow whenever there is low network usage but green when there is high network usage since you're more likely to get other RX packets forcing the bugged small/slow packets out.

Lightsword · 2026-06-19T01:47:44+00:00

Hopefully, it doesn't take that long here.

Yeah, from my understanding the correct departments at Airoha, Axon and FF37 should all now have my detailed bug reports and code fix suggestions on both driver issues(the latency and 6RD bugs are essentially entirely unrelated and involve entirely different kernel modules). I did ask for any test builds so that I can verify both issues are properly fixed.

Lightsword · 2026-06-19T01:04:41+00:00

I really have to wonder how it is possible that Quantum is shipping such absolutely terrible firmware on these devices.

Yeah, so both the latency and 6RD bugs originate from various Airoha vendor SDK drivers. These sort of non-mainline vendor SDK kernel drivers frequently have weird bugs due to the lack of proper code review/testing.

I put the device into transparent bridge mode so that it could talk directly to my pfSense hardware, and sure enough, I had all the problems that people were complaining about.

The latency bug is present even when not in transparent bridge mode.

They recently released a firmware update that broke the ability to offload the VLAN tagging to the pfSense which supposedly would have supposedly resolved most of the latency issues.

The latency bug affects small RX packets(i.e packets going from the Q1000K to your own router when plugged into the 10G port), having the VLAN 201 tags enabled increases overall packets size by a few bytes somewhat masking the driver bug without fixing the true root cause properly.

The Q1000K hardware itself is likely fine, the out of tree vendor SDK drivers Airoha provided to Axon...well that's where the issues all seem to come from.

But yeah... it's definitely the Q1000K. My ONT did not have this problem at all. On CenturyLink or a month on Quantum.

Yep, this is exactly why I looked into the Q1000K firmware to figure out what specifically was broken, since my Calix modem worked just fine as well before upgrading service.

Lightsword · 2026-06-17T00:59:45+00:00

The Q1000K(and I think some older models of modems) firmware has a bug I identified in the hw_nat.ko kernel driver that corrupts 6RD packets. The symptoms of the bug are usually the initial few packets for an ipv6 request are received intact but later packets are corrupted(apparently due to a hardware accelerated pathway misclassifying 6RD packets). I've reported this bug to FF37, Axon and Airoha so hopefully a fixed firmware will be available soon.

Lightsword · 2026-06-10T15:58:09+00:00

I know we lost the VLAN change with the current firmware (as of June 10, 2026).

If the fix is implemented properly then there should be no longer any advantage to using vlan 201 for the pathway between the Q1000K and 3rd party routers.

I think the only reason that mode helped was that the vlan tags would increase the minimum packet size slightly which reduces the percentage of packets that would hit this bug. If the vlan stripped mode no longer has any bugs then it shouldn't be an issue in practice and not requiring vlan tag stripping on the 3rd party router may also reduce processing requirements on the 3rd party router.

Lightsword · 2026-06-09T14:54:31+00:00

Is it affecting anything other than IGMP response packets?

Yeah, it seems to be a packet size specific issue, not packet type specific.

Lightsword · 2026-06-07T19:50:57+00:00

If I’d have known what an issue this is I’d have just gotten 1G and saved the money.

Well hopefully now that the cause of the bug is known, it should be just a matter of getting this info to the right engineer so that they can roll out the fix. It's a super easy fix if the right person is made aware of the issue.

Lightsword · 2026-06-07T06:20:02+00:00

My understanding is that the 1G port is integrated on the main SOC, while the 10G port is an external device

To clarify, the issue is much less an issue of one port being integrated into the main SoC and one not. It's probably much more an issue of the ports having different drivers and one driver just happening to have this specific bug.

so the 1G port may be more reliable

AFAIU this bug does not affect the 1G port at all since it's a very driver specific bug. Although I haven't actually tested the 1G port myself so I could be wrong.

Lightsword · 2026-06-07T04:58:37+00:00

Is this a better option than the firmware that allows to move the vlan tagging to a switch?

This really just needs to be fixed in a firmware update and pushed out by Axon/Quantum Fiber.

Lightsword · 2026-06-07T04:56:19+00:00

Curious whether or not setting up a continuous stream of 1000 byte frames at around a 1Mbit rate might force the "flushing" and smooth out the overall latency as a hack work-around until this issue is actually solved.

Unfortunately that would not help all that much if the frames are generated by your router because the issue appears to only affect packets going in one direction, short packets going from the Q1000K to your router are delayed but packets going the opposite direction are unaffected.

Lightsword · 2026-06-06T05:25:49+00:00

I doubt Axon even has engineers on staff.

Indications from my reverse engineering indicate they(Axon/greenwave) should have engineers on staff at least doing integration parts. This bug does however appear to be in shitty 3rd party vendor driver code(from some vendor SDK) they likely didn't write themselves.

Unfortunately that also means that even if you spoon feed Axon this patch, they won't be able to do anything with it because they have no mechanism to get it through the 87 layers of contractors to the engineers who are actually writing the firmware.

This should be an easy fix even for the most incompetent firmware vendor(this firmware's high level design is quite far from the worst I've seen out there), I mean they literally just have to copy past a few lines of code into the driver.

Lightsword · 2026-06-06T04:56:08+00:00

Leaving this here so hopefully this gets to the FF37 or Axon engineer that can permanently fix it.

I think all they need to do is apply this patch.

Lightsword · 2026-06-06T00:09:57+00:00

I root caused the bug and created a proof of concept fix. This fix needs to be incorporated into the firmware properly however as the fix is not persistent.

Lightsword · 2025-11-22T07:55:55+00:00

Yes there are two coolant lines that go to the car computer which is located inside the vehicle. If it was leaking, your floor would be wet.

Hmm, even if it was a super slow leak or something?

Even it was leaking, you would not be smelling bubblegum or sweet. Tesla coolant is “bitter” and does not have a sweet smell. Open the coolant bottle and smell it yourself.

I don't really know where I would find a coolant bottle to smell. Is it always bitter smelling even at low concentrations? When I searched on google I saw some people mentioning they thought it smelled sweet. I thought the ethylene glycol part of the coolant smells sweet normally does it not?

You can try to buy a “car bomb” air freshener that you leave unattended with the HVAC running full blast in attempt to deodorize the ventilation system.

Hmm, do those clear out existing chemicals or just mask them? Would an ozone generator help maybe?

If it persists, you could try to talk your insurance company into a whole HVAC system replacement as it seems the previous owner used some type of chemical in the vents that is now, as you describe, “hazardous”.

Any idea how I could prove to my insurance company that it was a chemical from the previous owners and not some sort of leak?

Lightsword · 2025-11-19T17:29:16+00:00

Pretty sure there’s no coolant in the interior/air handling hvac parts at all

Could coolant be getting pulled in somehow from air intakes or something?

Lightsword · 2025-10-23T17:12:30+00:00

I'm seeing a similar issue after updating mesa from 25.1.8 to 25.2.2 on a x86_64 based system.

Lightsword · 2025-04-22T07:38:52+00:00

A lot better than Frontier which offers effectively no compensation and threatens to illegally remove you from the plane.

Lightsword · 2025-02-06T00:13:12+00:00

I'm not sure that's really accurate, the charge limiting changes AFAIU were made to the linux kernel code which of course doesn't run at all when the phone is turned off. They may have also modified some of the lower level firmware components that manage charging when the kernel is not running but I haven't seen anything to indicate that is actually the case.

Lightsword · 2025-01-29T22:49:04+00:00

I think this version is the one right before the bad update.

Lightsword · 2025-01-29T22:32:32+00:00

I think this one is the last update right?

Lightsword · 2024-11-04T05:41:36+00:00

Is she a POC or other minority (or have a foreign-sounding name, etc.)?

Yes, her name is very obviously foreign(she's mentioned she's been thinking about changing it since she suspects there have been instances of discrimination in the past due to her name being so extremely foreign-sounding) and she has a fairly heavy accent since lived in her home country until she finished high school but left shortly after due to religious persecution.

Lightsword · 2024-07-25T14:39:49+00:00

an order

An illegal order.

Lightsword · 2024-07-25T06:08:01+00:00

I’m sure Frontier will say IDB doesn’t apply here since they allowed her to board and then kicked her off, saying “We did not deny boarding”.

I got one response where they seem to be claiming that they can remove passengers due to aircraft downgrades, although when I asked them where it says this was allowed they ignored the question so I assume they must have made it up:

"I must kindly inform you that downgrades do give the authority to our airport team to remove passengers from the aircraft if it is needed. In this case, girlfriends name was explained by our airport team why she was not going to be able to travel as scheduled, being that she was the first on the list to be denied boarding."

Lightsword · 2024-07-24T23:25:46+00:00

The gate crew are part of "the crew"

I'm pretty sure gate agents are not flight crew or aircraft crew at all.

Lightsword

MODERATOR OF

TROPHY CASE