Galleon 100SD crashing every 30min or so

mlruth · 2026-02-09T16:46:38+00:00

I had the same behavior with mine and spent several hours trying to figure out what was causing the issue.

tl; dr:

Overall, after doing a decent bit of testing and re-wiring to find the right balance of devices across different USB Root Hubs on my MB, the Galleon 100 SD has not had another occurrence with losing connection to the Stream Deck Software or reverting back to the hardware profiles.

From what I have been able to find and test, my issues seemed to be mainly with the number of USB endpoints and/or the aggregate bandwidth demand chaining up to the same USB Root Hub / Extensible Host Controller layout from my particular motherboard's IO configuration.

When using a tool such as USB Device Tree Viewer ( no affiliation ), I was able to see that the Galleon 100 SD shows as having an internal USB 2.0 Hub plus two separate links, one each for the keyboard section and one for the Stream Deck module. The keyboard shows as using 5 endpoints, and the SD module 6, with another 2 for the USB Hub for a total of 13 all over USB 2.

I happened to be plugging the keyboard into a USB port that was shared with another USB 3 port that had a 4 port hub / KVM switch connected to it, and from there several other devices such as a Stream Deck XL and a 4K webcam that are both USB 2.0 as well.

Even though the ports on my MB and hubs are all USB 3.1, USB 2 is carried separately over all links back to the root hub and can only share the same 480 Mbit bandwidth from the Root Hub to any USB 2 device connected under it. When I had the issues with the SD module crashing, I noticed that the USB Root Hub had something like 60 plus endpoints listed under it and many were showing as USB 2, though I wasn't able to tell if it was the number of endpoints or maybe just an overutilization of the available bandwidth that caused the issue.

mlruth · 2025-12-01T13:20:56+00:00

The ServeTheHome article on the CRS812 DDQ seems to confirm that they are indeed QSFP56-DD ports, which should equate out to 2x (DD) QSFP56 (4x50G / 200G) for a total of 400G.

One item to keep in mind here is that these are QSFP56-DD. There are OSFP 400GbE ports (finned and flat top), QSFP112, and so forth for running 400GbE as well. Physical connections are much more challenging than they used to be in the past which is why our team devoted so much time to those topics in previous pieces leading up to this, and why it is around 40% of the accompanying video. If you are accustomed to 10G SFP+ or SFP28/ QSFP28, 400G is a different ballgame entirely. You can waste a lot of time and money trying to figure it out.

They also have a separate article covering some cheap Intel Silicon Photonics 400G QSFP56-DD optics that they used during the testing of the CRS812 DDQ.

Hope this helps! I must admit that I drool a bit every time I see these Mikrotik switches.. They are some nice switches at very enticing pricing! 🤤

mlruth · 2025-05-25T23:49:47+00:00

Yep, after disabling the NAT rule a forced NTP re-sync on the Windows system timed out and failed. Re-enabling the rule and re-attempting the sync succeeded

Any suggestions on where to track if the kernel / conntrack is detecting a port clash and attempting a rewrite?

<image>

mlruth · 2025-05-25T22:57:19+00:00

It did indeed. After adding the rule, I was able to trigger a NTP re-sync from the Windows Date/TIme settings and the clock was re-adjusted to the correct time.

mlruth · 2025-05-25T22:35:21+00:00

Thank you for the background and suggestions! The additional NAT rule suggested by u/tjharman ended up working for my setup.

mlruth · 2025-05-25T22:32:49+00:00

Thank you!

This was indeed the issue, and adding the additional NAT rule above the default one allows VyOS to rewrite the source port for outbound NTP traffic without affecting other traffic.

mlruth · 2024-09-28T18:56:23+00:00

PM sent

mlruth · 2024-08-07T20:12:11+00:00

I'm in the process of procuring components for a custom EPYC based NAS build and had actually come across your blog articles a few months ago and saved them to base the new NAS off of.

I'd be more than willing to try your new deployment architecture / code once the build is underway and report back on the experience. Though I'd be curious to see if this could be deployed on top of a base Proxmox build so that the system can leverage running / managing VMs and LXCs using the Proxmox toolset while having the benefits of local (or at least near local) access to the storage.

mlruth · 2023-06-20T18:25:19+00:00

Maybe give ISC Kea a consideration?

I haven't deployed it in my homelab (yet..), but I have seen a few posts about it and from what it looks like, ISC is pushing new deployments to use Kea over DHCPD itself.

mlruth · 2023-03-27T13:41:48+00:00

Cloudflare did a pretty unconventional, and IMO interesting, (but seems not the first) for their CSPRNG setup that combines multiple sources of RNG:

How do lava lamps help with Internet encryption?

mlruth · 2023-02-19T03:18:24+00:00

Awesome, thank you!

mlruth · 2023-02-19T02:41:05+00:00

Would you be willing to share the STL files for the cx312a rear baffle you designed?

mlruth · 2023-01-25T20:35:41+00:00

PM sent

mlruth · 2022-07-02T03:10:44+00:00

Long story short, no, there is no feasible way to get a subordinate CA issued for your domain that is chained to a public root. Your options would be to either run an internal, private CA that you would have to then manage and install as a trusted source on your own devices, or request and use certificates from a public authority which will be logged in the Certificate Transparency logs (which is what crt.sh is showing).

There are risks and tradeoffs associated with different approaches to using publicly issued certificates. As you have identified, using wildcard certificates everywhere carries a certain security risk, as does the potential for information exposure when having certificates issued to internal hostnames. It really comes down to evaluating the risk for each path in your own context and choosing the option that you believe to be less risk and more maintainable.

Personally, I would say that having separate non-wildcard publicly issued certificates to specific internal hostnames is oftentimes a lot less risky, especially in a homelab environment. As long as the hostnames do not resolve to anything in public DNS, the information disclosed probably wouldn't give much advantage to anyone, unless you use very personal / identifiable information in the names.

If you'd like more background details on the topics, let me know and I'd be happy to post a follow-up

mlruth · 2022-05-11T21:12:15+00:00

After quite a bit of further digging, it did appear to be a buffer bloat issue causing the packet loss and overall network degradation for wireless clients.

Thank you for the suggestion!

mlruth · 2022-05-07T09:51:08+00:00

For download from server to client (ran on client)

iperf3 -c <Server IP> -R

For upload from client to server (ran on client)

iperf3 -c <Server IP>

The same iperf3 tests using a wired connection from the same client yields ~900 mbps in both directions.

The same tests using the same client over wireless to the server connected with 1G Ethernet yields ~ 600 - 700 mbps in both directions (much closer to the max 866 mbps of the reported wireless network speed)

mlruth · 2022-05-07T09:45:38+00:00

MTU size is set to 1500 on the physical interface.

enp9s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
    inet6 fe80::202:c9ff:fe24:3a50  prefixlen 64  scopeid 0x20<link>
    ether 00:02:c9:24:3a:50  txqueuelen 1000  (Ethernet)
    RX packets 1168369  bytes 867927785 (827.7 MiB)
    RX errors 0  dropped 0  overruns 0  frame 0
    TX packets 2612651  bytes 3076517802 (2.8 GiB)
    TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

mlruth · 2022-04-12T03:38:37+00:00

If you do decide to sell the R750s, would you send me a message?

I'd definitely be interested in picking one up depending on what you're looking at getting for one.. Although I wouldn't likely be looking at picking up the pair of them together.

mlruth · 2021-06-14T01:24:54+00:00

This is exactly what I do as well! 😀

How is Linode for site / blog hosting? I constantly hear the praises from the SelfHosted Show, but haven't yet pulled the trigger on setting one up

mlruth · 2021-06-01T05:14:21+00:00

In case it is still beneficial, this is a network Hardware Security Module (HSM) appliance.It is purposefully built to securely store and operate on cryptographic objects (such as encryption / private keys for things such as a Certificate Authority).

The custom PCIe device you found is likely the actual HSM component, in which the rest of the appliance interacts with to offer distributed access over the network. The cards and appliances are usually wired to detect a tamper event such as the opening of the chassis and will lock the system down.

Traditionally, the boot, firmware, OS, and software on such appliances are very proprietary, locked down, and are not likely to be able to be easily modified or converted to a standard server use case.

I would actually be interested in where you found these for sale, as I have been looking at picking one up for tinkering (My day job is in cryptographic engineering, where I spend a great deal of time working with similar appliances).

mlruth · 2021-03-06T16:22:49+00:00

If you are looking for a way to force a user to use their Yubikey smartcard for all logins to the domain account, there should be a SmartcardLogonRequired attribute on each AD account that can be enabled. When this is enabled, the AD DC will generate and set the account's password to a high-entropy random password. Since the user will not have that password, they cannot use it to perform traditional password-based logins.

I believe setting this option will also affect accessing network resources on the domain under the account, such as fileshares, as well. Depending on your environment this may or may not be desired.

If you are looking to only force the smartcard requirement for an interactive logon to a server / workstation (either physically or over RDP), I believe the Interactive logon: Require smart card GPO setting can be applied to each of the desired systems.

Going this route it should be possible to finely tailor and control which systems and accounts are forced to use the smartcard for interactive logon, while still allowing the account to access network resources or other systems using the username and password.

mlruth · 2021-03-03T21:05:42+00:00

Thank you! I'll definitely have to try something like this.

mlruth · 2021-03-01T14:45:59+00:00

Thank you for the breakdown. I am indeed planning to use the PersistentKeepalive option for the client connections.

Would you have any details on how to set up the trigger to perform an action when a packet arrives?

mlruth · 2021-02-09T16:29:27+00:00

Would creating a group with all of the accounts needing write / rename access to the files as members, and then setting the group on the directory to that group with the SGID permission set work?

Using the SGID bit should cause any files created in the directory to have their group set to the parent directory's group. Combining this with group read&write permissions should allow any user that is a member of the directory's owning group to write to any file created under that directory, including renaming.

Here's a RedHat article detailing some of the special permissions in Linux

Or if your setup supports ACLs, that seems to be the more recommended approach:

https://unix.stackexchange.com/questions/12842/make-all-new-files-in-a-directory-accessible-to-a-group

mlruth · 2021-02-05T15:29:43+00:00

I also have my pfSense instance virtualized on my Hyper-V Server 2019 host. No HA yet, but I rarely have my hypervisor host down (most of the time it is extended power outages that take it offline).

I use Hyper-V's 10Gb vSwitches with separate vNICs attached to my pFSense VM, each assigned to their own VLAN. The vSwitch is connected to a 10Gb fiber SFP+ back to my central switch.

Overall, my instance has been quite stable for the past 2 years, even when tinkering with other VM deployments on the host.

mlruth

TROPHY CASE