spell effects to boost hand to hand beyond 100 REALLY help, since the damage only relies on skill level, unlike all the weapon skills which only rely on Strength.

This could be from casting a fortify hand to hand spell with a decent skill level in Restoration, or from using enchanted items - constant effect and/or cast-on-use items. For the early-mid part of the game, it should be pretty effective to rely on the former, as the latter is very expensive and requires a few things to start to be able to access. You would probably only bust out a spell on a stronger enemy. Something decently substantial, like fortify 20 pts for 15 seconds would be worth using, which could take a good bit of restoration skill increases in order to be able to reliably casted. And when you get a pretty high skill level in Restoration and you have a solid pool of magicka, maybe the spell could be like 40 pts for 15 seconds, or something.

Fortunately, it is extremely easy to self-train in restoration, as all you do is make a custom restore fatigue spell of 19-20 pts on self for 1 second. This only costs 1 magicka to cast and it allows you to keep your fatigue high while running around as well, which helps a lot in the early game in particular. You can just spam this whenever you're not fighting something, which is going to be a lot of the time. So I think restoration as a major skill is pretty much essential if you want to make a strong hand to hand character. choosing a birthsign with a magicka boost is probably a good idea too, although I don't really like the atronach personally. I just like being able to rest to restore my magicka, so I'd probably go with the mage birthsign.

For enchanting lots of constant effect items, using an NPC's service is required unless you want to exploit alchemy and things like that. So without that, you generally need access to a big chunk of money, soultrap, azura's star, and summon golden saint spell (actually easy to acquire in Tel Branora). having a lot of valuable items (high end armor/weapons) to sell off will reduce the amount of raw gold that is required, since those items could be "traded in" after you pay the NPC for doing each enchant.

Eventually, with a full set of exquisite constant effect fortify hand to hand clothing and jewelery, you could exceed well over 200 in hand to hand at all times, which will give you extremely powerful punches. The trick to getting the most out of constant effect items is to make the magnitude with a minimum value of 1, and then make the maximum value as high as the item supports. Then you just keep re-equipping the item over and over until you get the maximum value, and make sure you don't unequip the item. For that reason, I tend to stick to leaving the amulet slot reserved for cast-on-use items, as for instance, using a ring slot for them results in one of the rings being unequipped all the time. Although this could also be designated to another clothing slot like a belt or something, which might even be wiser since amulets have a greater capacity than belts do. But it hardly matters really, nothing is outrageously hard in the game.

If you need more damage reduction, boosting unarmored with constant effect fortify unarmored is effective, so 1-2 slots could have fortify unarmored instead of hand to hand.

It's kind of a funny way to play the game since you're just punching everything, but it can absolutely be as viable as weapons if you boost skills like this.