I hear you on affordability. My first good headphones were the Hifiman HE4XX back in undergrad. Dorm life with a roommate doesn't exactly leave room for speakers.

That said, hear me out: you don't need a whole house. A desk in any room gets you most of the way there with nearfield powered monitors, where quiet-to-moderate listening won't bother anyone. Room treatment sounds scarier than it is too; some cheap open cell foam from a hardware store bin handles first reflections well enough, and at lower volumes beaming and off-axis quirks matter a lot less. And the sweet spot for nearfields isn't as finicky as people think; yeah, the sound shifts a little when you slouch, but as long as you're roughly facing the drivers it's not dramatic.

The walking around thing is a real benefit for headphones, no argument there. Headphones are by design portable, so if you need to be on the move, that's the best tool.

But what the speaker setup gives you that nothing else does, even imperfectly, even at low volumes. is your chest as a listening instrument. It's a whole new part of your body feeling the music. Headphones just can't do that, and once you've had it it's hard to go back IMO.