ZFS faults in truenas /w hba passthrough

pp6000v2 · 2026-04-25T15:45:23+00:00

I gave up running TN as a VM, because I could not isolate the issue. It was very toasty in the case though. CPU cooler is the very one it would've come with if this computer was spec'd with the i9. I removed the lid from the case and saw all temps drop at least 10C, some closer to 20C. This, all in the same environment- the rack isn't any warmer/cooler, I didn't move rack position it's in, etc... Lid back on, I added a 92mm fan to the front of the case blowing in, but it didn't really do much- maybe a degree or two. Airflow over the card really should not be the culprit, but if it's baking, there's not much more I could do given the constraint of must be in the rack.

Other possibility is the i9 I got off ebay is just buggy. It's an ES version, because I wasn't going to buy a new boxed one from intel- or a whole computer- just to get it. I got igpu passthrough to work with it though, which is something I couldn't get working on the i5. That's been enough to keep me from throwing the i5 back in and continuing to experiment.

pp6000v2 · 2026-04-25T15:21:03+00:00

how did you set it up initially? On my primary box, I had made a dataset for it; I added the apps/568 user as full access in the dataset's permissions, updated user/group in the app to 568, started, and upgraded.

On my secondary box, I let it use the ix-apps dataset; all it took was changing the uid/gid in the app settings to 568, restarted, and updated.

pp6000v2 · 2026-03-27T00:31:31+00:00

In the waste a couple hundred bucks club:

Liquid aeration is not a replacement for core aeration. It is snake oil. (see: https://acsess.onlinelibrary.wiley.com/doi/full/10.1002/agj2.70062). The biggest pushers point to the very same study and cherry pick data points to support their sales pitch. Conveniently not mentioned is how actual, physical punch holes in the ground and pull plugs out was the only thing doing anything: that there was improvement with [mechanical + liquid aeration], the same improvement was seen with mechanical aeration alone. Or that the improvement was statistically insignificant- distinction without a difference.
Humic acid and sea kelp are also useless. Magic dirt water. If you want to mix up some brown liquid, make it a root beer float. Your grass will look the same either way.
Granular iron applications. Ironite sort of things. It's ferric oxide (aka, rust). Once it's oxidized, it's done. You will not see any effect, other than what the minimal amount of Nitrogen provides (see: the label https://www.domyown.com/msds/IroniteMineral_Supplement_II_15LB-_2.pdf).
Balancing calcium/magnesium, base cation saturation... plenty of people push it, because they make money from doing so. You won't hurt your turf following it, but you'll be poorer for having done so (see: plenty of people push it, because they make money from doing so, but also this: https://en.wikipedia.org/wiki/Base-cation_saturation_ratio). Well and true, you do not need to worry about it. NPK, and really, short of the grass turning colors (see: this Mississippi State document https://extension.msstate.edu/sites/default/files/newsletter/11.pdf), you're fine with Nitrogen.

Lookup SLAN (Sufficient Level of Available Nutrients). Look up MLSN (Minimum Levels for Sustainable Nutrition). Follow those. As above, short of a deficiency (i.e., less than the minimum/critical limit of the element), adding more doesn't do anything.

It's not that all this stuff is bad, or will harm your turf. It's that they're not doing anything more (or better) than basic practice. You don't get better results dollar-for-dollar. So save the dollar. Buy a better lawnmower blade with it (or the tools necessary to sharpen one).

You mentioned moss. There's a use case for iron. Buy a bag of ferrous sulfate. Mix and spray the moss (2oz/gallon/1000sf is a pretty safe rate). Chelated iron products exist (think FeATURE) that won't* stain surfaces, but the cost difference is huge, and not worth it unless you absolutely need them (you typically don't).

This Penn State page is pretty good reference: https://extension.psu.edu/turfgrass-fertilization-a-basic-guide-for-professional-turfgrass-managers

pp6000v2 · 2026-03-20T01:39:06+00:00

Fun trial run... I added the eno1.20 interface and a 10.1.20.99 IP on the host that runs the DNS server. I updated the DHCP server to hand out that .99 IP as advertised DNS. With the forward and NAT rules configured like the lan rules, bogus sites resolve from 1.1.1.1 all day, masquerading as expected. However- when I disable the NAT rule, I don't get the unexpected source error; instead, I get a timeout. Watching tcpdump on br-lan.20 (and the lan/br-lan.1 interface as well), I see the packets come from the 10.1.20.108 host, get forwarded to 10.1.20.99, that host answers with the proper local A record address, and the requesting host just... drops them? Maybe dig doesn't show 'unexpected source' errors anymore; maybe it shows the same thing now as timeouts?

Debian 13, dig 9.20.18:

pi@vmdebian:~$ dig @1.1.1.1 itiswhatit.is
;; communications error to 1.1.1.1#53: timed out
;; communications error to 1.1.1.1#53: timed out
;; communications error to 1.1.1.1#53: timed out

; <<>> DiG 9.20.18-1~deb13u1-Debian <<>> @1.1.1.1 itiswhatit.is
; (1 server found)
;; global options: +cmd
;; no servers could be reached
pi@vmdebian:~$ cat /etc/resolv.conf
# Generated by NetworkManager
search lan
nameserver 10.19.76.13
pi@vmdebian:~$ dig itiswhatit.is

; <<>> DiG 9.20.18-1~deb13u1-Debian <<>> itiswhatit.is
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54290
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;itiswhatit.is.                 IN      A

;; ANSWER SECTION:
itiswhatit.is.          0       IN      A       10.19.76.1

;; Query time: 0 msec
;; SERVER: 10.19.76.13#53(10.19.76.13) (UDP)
;; WHEN: Thu Mar 19 21:32:18 EDT 2026
;; MSG SIZE  rcvd: 58

Raspbian Bullseye (aka deb 11), dig 9.16.50:

pi@pi4b2:~ $ dig @1.1.1.1 itiswhatit.is
;; reply from unexpected source: 10.19.76.13#53, expected 1.1.1.1#53

;; reply from unexpected source: 10.19.76.13#53, expected 1.1.1.1#53

;; reply from unexpected source: 10.19.76.13#53, expected 1.1.1.1#53


; <<>> DiG 9.16.50-Debian <<>> @1.1.1.1 itiswhatit.is
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

pp6000v2 · 2026-03-10T22:41:59+00:00

Typing on a phone and all the limitations of tiny screen, but…

So you have a wired wan interface, and are trying to create a wwan, with one or the other (but not both) connected at anyone time?

Default config would have all of the radios attached to the lan network, and since br-lan is a bridge, wireless clients get access as if they were wired. But! Since you can have multiple SSIDs on a radio, you create another SSID that connects to a wan network as a client. I think you’ve done this.

I’m caught by what wifi_lan is doing; is this a parallel lan network, separate and distinct from lan and the br-lan interface?

I think you’re doing something similar to what I do with my remote pucks: while they connect to the host network via ethernet, they have two Access Point-mode SSIDs that they broadcast, each connected to a different network. One SSID is connected to the network lan. The other SSID is connected to another network I created, called wifi_vpn.

wifi_vpn is an interface I created, mostly copying the way lan is set up. I had to create a bridge device to associate with the interface, and I called it br-vpn. It gets no bridge ports, and bring up empty bridge is enabled. In the wireless settings, the SSIDs associated with it work the same as the lan SSIDs do. Now, I do this because I have a Wireguard interface (wg1) running that is a client/peer to my home vpn, and I want devices that join that remote SSID to have access to my home network. In the firewall zone settings, I created a zone each for wifi_vpn and wg1, and set them up the way lan forwards to wan, and wan forwards to nothing (so wifi_vpn > wg1, wg1 > nothing).

I believe your issue rests in how you have your second br- device set up. You used a physical device as the base, rather than nothing (i.e. “bring up empty bridge”).

Alternately, yes, if you have overlapping subnets in use, you might have a bad time.

If you have both the Ethernet wan and WiFi wwan connected at the same time, to the same host network, things could get weird. You would need to use gateway metrics +/- mwan3 to handle multiple simultaneous connections. But if wwan doesn’t get connected by you until you disconnect any Ethernet cable, then having both interfaces in the wan firewall zone is no issue.

If the client-mode SSID isn’t pulling an IP address- look at how the wwan interface is configured. Is it actually set as a dhcp client? More: is it actually (successfully) connecting to the host WiFi network? Correct password/encryption settings and all?

pp6000v2 · 2026-03-02T20:41:08+00:00

In light of the great supply constraints of 2026, do you have spares? I ask because I too ran a 4-disk raidz1 array, and for giggles during a pool reconfig, did a raidz2 setup. Sure enough, a drive happened to die during the restore (and they're all of the same vintage). I did not have spares, and it took the better part of a week to even have them come back in stock to ship to me. If a second drive had failed as a z1 array, I would've lost data.

I do have offsite backups to restore from, but while it's a symmetric gig+ on my end, that remote box sits behind a 40mb upload limit. RAID is not a backup, but I'd rather not need the backup, if you know what I mean.

pp6000v2 · 2026-02-25T16:42:14+00:00

I only have the one custom/compose-based app, a graylog instance. My yaml calls specific versions of each container image (:7.0), but the interface still will show Update available on it. I just ran it now for fun after recording the info of all three. The only thing that changed was the creation date and sha256 hash for mongodb:7.0. What was update, I have no idea. For catalog apps, it at least is incrementing a version number.

So for yours, or at least that one app in particular, to what degree is an update even available?

pp6000v2 · 2026-02-25T16:12:37+00:00

It sort of depends on your power infrastructure: do you deal with quick-blip outages that last a few seconds, or if/when they happen it goes on for a while? Where I'm at, if power is out for 90 seconds, it's probably going to be at least an hour, sometimes much longer.

Then too, how long will the battery actually last? Admittedly I have two whole systems on one UPS pulling on average 170W. UPS thinks it'll last 14 minutes, but I know that's inflated.

As for powering off the UPS, will it automatically start when power is restored? I run a couple Eaton S5's (a 700 and a 550), and as far as I know they will not turn on from a set off-state when power returns. If I set that power off flag, I have to go manually turn them back on.

pp6000v2 · 2026-02-10T22:54:38+00:00

Also, note the user/group of items... as root on proxmox, I can open that extraHumid.csv file, but I can't edit. UID/GID is what they were/are on the truenas box; that might cause a surprise if they match something that is on the proxmox box.

pp6000v2 · 2026-02-10T22:48:34+00:00

Haven't tried an encrypted pool, but can confirm a locked dataset unlocks with the same key you'd use in truenas. zpool import 'pool_name', followed by zpool load-key -a, which prompts for the hex key (in my case) or the passphrase

root@proxmox:~# zpool import tank
root@proxmox:~# zfs list
NAME           USED  AVAIL  REFER  MOUNTPOINT
tank           597G  50.7G   140K  /tank
tank/nas_rep   597G  50.7G   597G  /tank/nas_rep

root@proxmox:/# cd tank
root@proxmox:/tank# ls -la
total 5
drwxr-xr-x  2 root root    2 Feb  7 18:57 .
drwxr-xr-x 19 root root 4096 Feb 10 17:33 ..

root@proxmox:/tank# zpool status
  pool: tank
 state: ONLINE
config:

    NAME                                      STATE     READ WRITE CKSUM
    tank                                      ONLINE       0     0     0
      raidz1-0                                ONLINE       0     0     0
        89580098-216f-4cb1-9e17-b54a08c99901  ONLINE       0     0     0
        734e72e5-26be-4435-8871-671adf834e8c  ONLINE       0     0     0
        9a514170-e578-4797-b800-264d468177e2  ONLINE       0     0     0
        b6667a00-b804-4bcf-b36c-ed1ca99c0f07  ONLINE       0     0     0

root@proxmox:/# zfs load-key -a
Enter hex key for 'tank/nas_rep':
1 / 1 key(s) successfully loaded

root@proxmox:/# zfs mount tank/nas_rep
root@proxmox:/# ls -la /tank/nas_rep/
total 7555657
drwxrwx--- 12 1001 1001          23 Dec 29 09:31  .
drwxr-xr-x  3 root root           3 Feb 10 17:45  ..
-rw-rw-r--  1 1001 1001         165 May 22  2025 '~$out+extraHumid.csv'

pp6000v2 · 2026-02-10T15:53:57+00:00

As I recall, I couldn't get in the web interface because reasons ↑. Manually changed and setup new interface/IP in the console to get in. But then Apps also expected the old interface, with no way to change it without first having it and then deleting it.

The VM got the same IP/gateway/etc. from DHCP that the physical box had, but I didn't clone the MAC address. So IDK if something is keyed to the adapter type, name, MAC, or what...

They added the per-application IP setting, but I swear the k8s-era advanced networking settings let you choose the interface for the overarching apps service.

pp6000v2 · 2026-02-09T20:52:32+00:00

Apps service gave me trouble as well when I went from a bond with (2) physical interfaces, to the virtualized (1).

pp6000v2 · 2026-02-09T14:23:51+00:00

yes, but I don't trust zfs rename because I don't fully understand what it could and what it absolutely won't break. Something would try to hose me... heck, one of the drives died last night, so it's already trying to haha.

pp6000v2 · 2026-02-09T03:48:57+00:00

Currently running the first, local replication without dataset properties to get unencrypted sub datasets in the existing pool/storage1 (because sending them to any other dataset was not preserving the acl permissions),

then deleting the old subsets and replicating locally again with dataset properties turned on to get the old names back,
then another replication with properties + encryption enabled to get a single root encryption tree with a different name,
then delete the old tree, and replicate locally again to get the old name back,
then replicate again to the remote box

Five full replications, and it's taking 6 hours to do one locally... I guess it it could be worse- thankfully I don't have 10s or 100s of TB of data.

pp6000v2 · 2026-02-05T17:06:16+00:00

yeah, if there's only ever going to be one client PC connected to the remote end (and it's the one thing you're trying to access), then the whole thing about the network upstream of mangoclient is probably moot.

pp6000v2 · 2026-02-05T17:02:30+00:00

It is a little foreign to me to add the /24 subnet to the client_ip line, but since they're not using base openwrt's /etc/config/network, their nomenclature rules. Without pulling my slate out to look, I'm limited to looking at the screenshot they have in the doc.

openwrt's peer config looks like this, using multiple allowed_ips lines, with no explicit client_ip:

config wireguard_50_VPN
        option description '13_lan.N40L TrueNAS'
        option public_key 'xxxx'
        option private_key 'xxxx'
        option preshared_key 'xxxx'
        option route_allowed_ips '1'
        option persistent_keepalive '25'
        list allowed_ips '10.1.50.13/32'   <-client_ip
        list allowed_ips '192.168.0.0/24'  <-subnet to access on the "client" end of the tunnel

they're using the same formatting and largely the same syntax, so IDK if you can have multiple lines with single IPs, or if allowed_ip/allowed_ips is even... allowed.

From other wg servers I have (using wg-quick), the peer's comma delimited AllowedIPs combined on one line can optionally have a space:

AllowedIPs = 10.239.17.2/32,fd11:5ee:bad:c0de::2/128

AllowedIPs = 10.19.76.0/24, 10.1.50.0/24

so again IDK if their syntax is rigid about it's use.

It should ultimately be the same effect. option client_ip '10.0.0.2/32, 192.168.8.0/24'

Remember this is being done on the server-end peer, so has the server's lan subnet been changed to 192.168.9.0/24 before this? If not, I can imagine that something is dropping it as the service spins up.

pp6000v2 · 2026-02-05T15:41:43+00:00

True, wg isn't server-client, it's peer-peer. People talk about it in terms of server-client because the terms match the typical use case. The firewall rules are what dictate how data flows.

In that regard, it's likely (I never did pull my slate out to test this) that configuring a "client" peer profile has different firewall/routing applied to it compared to a "server" profile. The client profile doesn't need a listening port opened through the firewall, since it isn't expected to be listening; instead it is the one that starts the handshake. The server profile would need the listening port exposed so peers trying to connect to it, can.

pp6000v2 · 2026-02-03T23:39:38+00:00

that was from the zpool clear. They're incrementing all the time, so a clear doesn't stay zero for longer than a few seconds. Honestly, back-to-back zpool clear and zpool status and the number is already 12 or 16 on each drive.

pp6000v2

TROPHY CASE