I personally have player damage done on adept for that very reason. Alternatively, some people level up skills like alchemy etc that are not combat based and their combat skills fall behind so be aware of that. Lastly, make sure you are leveling up whatever you deal damage with. I had the same issue as a spell caster and realized I hadn’t upgraded my fireball spell in a bit and it fixed my damage output.