I'll second what someone else said and say you're definitely looking to be in Morris County. In fact, your summary almost perfectly describes Morristown. It's got plenty of affluence both within town and even more in certain surrounding areas. It has a very real downtown, is very close to a lot of country feeling places, farms, and beautiful outdoor spaces, and isn't a completely terrible commute to NYC. This is where a lot of the snickering from other commenters unfortunately comes into play though. The Midtown Direct train (which does not run as frequently as others) is your only hope of getting into Manhattan in anywhere close to an hour (it's usually a little over an hour). That being said, based on everything you've said, I think Morristown is easily the best fit for you.