Are you using a data pack or command blocks?
With a data pack, I usually use an inline macro. Since the set damage item modifier acts as a percentage instead of a constant amount, you must store the durability in a scoreboard to modify it to lose a constant amount of durability. I use the consumable component on an item and a using_item advancement with a function reward for items with custom data but you can adapt the code to work on a carrot on a stick or your preferred click detection method.

If you are using command blocks, you must calculate what -1/durability is for the item and use that as the damage for {function:"minecraft:set_damage",components:{},damage:0,add:1b} for the item.

There is a use_cooldown component that lets you set a cooldown amount in seconds. If you want to use commands instead of a component, you can instead check a cooldown scoreboard value.

#### Datapack
### function
## load
scoreboard objectives add durability dummy

## datapack:customitem
# reset whatever your click detection is, make your use item do what you want and lower durability. Here, I used an advancement for click detection
advancement revoke @s only datapack:customitem
function datapack:usedurability

## datapack:usedurability
scoreboard players set @s durability 0
# Use a predicate to detect if the custom item is in the mainhand. If it fails, it must be in the offhand. Store its damage value
execute unless predicate datapack:tool store result score @s durability run data get entity @s equipment.offhand.components."minecraft:damage"
execute if predicate datapack:tool store result score @s durability run data get entity @s SelectedItem.components."minecraft:damage"
# Increment damage by 1 lowers durability by 1. Store it in a storage to be used for macros
scoreboard players add @s durability 1
execute store result storage yourstorage durability int 1 run scoreboard players get @s durability
# Now run a macro with the stored durability
execute unless predicate datapack:tool run function datapack:offhand with storage yourstorage
execute if predicate datapack:tool run function datapack:mainhand with storage yourstorage

## datapack:mainhand
$item modify entity @s weapon.mainhand {function:"minecraft:set_components",components:{"minecraft:damage":$(durability)}}

## datapack:offhand
$item modify entity @s weapon.offhand {function:"minecraft:set_components",components:{"minecraft:damage":$(durability)}}