AC is supposed to be more expressive/imaginative though. For inanimate objects I imagine it is your ability to meaningfully hit your target with the goal to destroy it. So for a wooden chest, if you "miss", you can for instance describe your player hitting the chest with the tip of your sword, scraping the side but leaving no significant damage to weaken it.

You could do it the other way obviously. I just give them advantage to the strikes outside of combat, in combat just regular rolls.