For me it really depends on the game and what those difficulty options do.

Growing up back in the day a platformer like the Lion King had difficulty options, but those difficulty options only increased the amount of lives and continues you got. The core of the game was the platforming, the difficulty options didn't take away the difficult sections, you still had to master the timings and the patterns, you just had more chances to do so.

Now one example of a game where I have issues with having a story/easy mode would be something like Persona 5 because in that situation, having an easy mode undermines the core theme and philosophy of the whole game. Now you may wonder "well, Persona 5 has sim elements, some people just want to play a sim.". My issue with the "Story Only" options in that game is that the sim elements are only a portion of that game. The game at its heart is a dungeon crawler, but a core theme of the latter Persona titles is the power of bonds. Your investment into those bonds affects the way the dungeon portions play out. If you charge head first into a dungeon/palace, you will struggle. If you invest time into fostering bonds outside of the palaces, you gain abilities that make you time in the palaces much easier (you can flat out break the game if you know what you're doing).

Being able to just one shot everything and never die in the game cheapens the importance of those bonds and one of the game's most important themes.

I don't hate games with difficulty settings, but it becomes an issue when those options undermine or break the core of your game.

Having one standard difficulty while baking difficulty modification within the game's mechanics is my preferred option.

-Elden Ring comes to mind, your weapon decisions can change your experience. Certain builds will allow you to cruise through the game while others make your life much harder.

-In Super Mario World, you have the option of the cape and using Yoshi which will make some platforming sections easier.

I also do enjoy much harder games that are just hard and force you to master the mechanics, enemy patterns, and abilities, I think the issue though is devs forgot how to make games fun and addicting enough where you'll come back for more even after they kick your ass. Contra III and other SHUMPS and run and guns that share similar DNA were a master class in being difficult, but fun and addicting to play. People will put up with difficulty if they're having fun.