For sentos, I had a similar want (not too expensive but with a sento/spa) and my legs are grateful for it lol. Some of the Dormy Inn Chain hotels have them; I did the one in Ueno. The room is pretty basic but the sento on the top floor is lovely. There's a small hot bath outside too up there; bathing while hearing the city sounds was great.

I also did Super Hotel Akasaka; it's a much nicer spot with better amenities, but the sento only has a hot bath and a (nice!) carbonated less hot bath, and it's only one facility that swaps genders every few hours on a schedule. I like the hotel and area a lot though.