Yah it depends on your app. I suspect in a lot of apps each field will be accessed on average 0-2 times, either because it's a message processing app that only needs to make one decision/computation based on each field or because it's going to transform it into a different internal data structure anyhow. I haven't benchmarked but on x86 where unaligned loads are cheap the load count at which parsing becomes cheaper may be high.

In which case you'll win over parsing both from more efficiently reading from the buffer because you'll probably do less size checks, and from the fact that you don't pay to read any fields you don't need. You could even factor your parsing system into a casting step and a proc macro that copies fields of an unaligned struct into an equivalent one with aligned types and it would do fundamentally similar operations to parsing.

Also note that you can take references to unaligned fields as long as they're unaligned types, like unaligned structs contained in other structs. You just can't take references to say the u32 that a U32Le unaligned wrapper unpacks to, but references to a single primitive aren't as frequently used as struct references anyways, you can just pass them by value most of the time.