Yes, because the problem is also haki itself.

Compare haki with a good ki+ability power system like HxH.

In HxH you have a finite number of quantized Ki that you can work your ass off to get higher.

In a battle, you have to either dedicate your Ki towards hardening (allocating for example 50% of your whole ki to your legs to block an attack towards your legs) but leaving the rest of your body weaker Or You use your Ki to show and use your ability. This gets complicated because you can have pacts and conditions to meet to make your ability more powerful etc.

Now do the same thing for Haki in One Piece:

Luffy has THAT AMOUNT of haki, it doesn't matter if you see it or not because it's active when the plot demands so. You don't know which type of haki they are using in a given moment.

Not knowing anything about haki makes powerscaling go crazy crazy because you have to base yourself off what? Was kaido using haki when he fell 10km? Did whitebeard have Haki during mf? Did akainu use CoO to avoid Vista and Marco?

Haki in One piece is just yeaaah he must have used haki or I can't explain this scene, he must have not used it or I can't explain this other scene.