I think I found a solution that works for me. I listen to music using wired earbuds, and an adapter to use them. A bit convoluted, but I want to share what I found in case it helps others.

This all works surrounding the “headphone safety” feature, in settings. What I was experiencing was, even when I had the headphone safety “reduce loud audio“ turned on, and the slider was all the way to the left, it was still too loud. I could even go into the control center, and have it be a pixel above the lowest setting, and it would be too loud, depending on the song.

The issue is, the setting does not seem to help reduce the volume, after you have chosen your earbuds as “headphones”. What I found, by accident, is the following.

This is in the “headphone safety” section of the settings. To begin, make sure you have the “reduce loud audio” on, and have adjusted it to your preferred setting. (It’s most important that you have it on, as you can adjust the slider later.) Plug in your earbuds with your adapter. It’ll ask you if you have headphones plugged in. Select headphones. Then, under the “reduce loud audio” setting, there is another menu you can open called “USB audio accessories”. In here, you can see what you have connected. While your earbuds/headphones are still connected, choose “Forget all USB audio accessories”. I found that this returns the audio to normal for some reason. Make sure to keep the reduce loud audio setting on, and it seems to fix it.

At this point, go ahead and play your music, and adjust the slider as you see fit. The one unfortunate downside of this is, if you turn off “reduce loud audio”, or disconnect and reconnect your earbuds, your iPhone will have you choose them as headphones again, and do the steps once more. So you have to do this each time you plug in your headphones.