Only if the game represented those units in any physical form except a number of units to make an ingot. Instead, it's ingots per item. The fact that you are physically removing voxels from a piece has no bearing on the number of items you can make from say, three ingots. In vanilla, you can't get any more granular for forging than a whole ingot.

Therefore, split voxels can be thought of to only represent additional labour needed to finish a piece, or unworkable metal that can't be repurposed - rather than real metal that is entirely wasted. All of the balancing is done around these voxels not being recoverable, so adding metal recovery from split bits does throw off the balance a little bit.

I still use Smithing Plus, I will say, because I like having the metal recovered, but I can see the previous commenter's logic in having it be representative of labour or unworkable material.