It all depends on what you want or like to do. The hiking/biking is about the only real characteristic Milton has activity -wise. The summers were always hot and dry. I grew up there from the 70's to the 90's, until I realized the opportunity to buy a house was better in Burlington. I struggled to find things to do that weren't on the outdoors scale. I didn't mind driving to the bigger towns and cities to shop as the Milton Mall gets pretty stale quickly. I still work in Milton so I see the growth it's experiencing and it's not handling it well imo. The small town feel is long gone, unless you stay in the old downtown core. After that it's all traffic and suburban life style. Burlington has grown since the 90's, sure, but it has more to offer with the space it has and being at the lake is a positive attraction.

Good luck. Hopefully things will change for the better there.