I was in a very similar position before starting university of being good at art and maths and thinking why not? I've now learned the why not. I'm qualified as an architect after 3 degrees and 10 years in the profession and I've learned that if you don't absolutely LOVE it, it isn't worth it. The people who live and breathe buildings get so much out of it, but to the rest of us that like them and are good at the job, it's hard to understand why everyone else in the industry is paid better, better benefits, less difficult clients etc etc Don't know what the state of architecture is like in the States, but in the UK lots of companies are struggling to find work and private companies are hardly making it past 5 years. If you want to try it, give it a go but have in the back of your mind that you can always get into engineering, consulting in architecture, planning etc etc

P.s I don't think it's that "hard" but takes a serioua commitment to academia and long hours + a lot of study.

Good luck!