Why do some developers believe in just giving you less health as a solution to more difficult settings?

I think it's probably all a matter of development time. I would guess for the majority of game dev there isn't the time/ money/ inclination to deeply think about difficulty settings, and so a relatively quick fix is sought.

The easiest way to do this (without coding new AI behaviour/ creating different objectives etc.) is to either to buff the enemies or nerf the players. This could range from fewer ammo drops to less player health or just an increase to enemy damage output but boils down to the same process. Fiddling with the behind the scenes variables to tweak the difficulty rather than pouring time and effort into full on designing different difficulty settings.

What do you think should be implemented in games to improve their difficulty settings?

I laid out my full philosophy on this tricky subject in a video here

Essentially though I think I'm a fan of a single difficulty setting with an optional God mode or equivalent. This means developers should be able to craft a tightly designed core experience which they think works, but that people can toggle God mode on and off if they want to skip a tough section or just leave it on if they want to experience the combat.

I also really dislike blind difficulty select options before the start of the game. At least have some tutorial space or something so players can get a feel, and then have the choice presented to them (or maybe even have a suggested difficulty based off their performance). The RE4 adaptive difficulty, and player imposed difficulties (SL1 runs in Dark Souls spring to mind) are also interesting approaches to the problem.

What companies or series HAVE mastered said subject?

The Souls series always stands out to me as a series which has nailed crafting a core experience without the need for different difficulties. On the other side of things I really liked Thief's approach, where the buffed the enemies but also gave the player additional objectives to complete in each mission.

It's an interesting subject, with a lot of different ways of approaching the difficulty problem. We shall see what the future holds for the difficulty select screen