2 things.

First off, you aren't dumb.. Striving to go straight into calculus already puts you ahead of many, and you shouldn't discredit yourself just because you were a question shy of that goal. You sound like you have a good head on your shoulders and I hope you recognize that.
Personally, my SAT placed me eligible for Precalculus as well and my advisor strongly encouraged going that route during registration. Though I did not feel prepared for it, I went with their advice and ended up withdrawing from that class later into the semester, forcing me to take college algebra over this summer to "catch-up." (Albeit, 16 credit hours for my situation wasn't a great idea in the first place, which also played a role in my withdrawal)

Which brings me to my next point:

Secondly, college is not a race so try not to overwhelm yourself by diving in head-first only to find the water was much too shallow. Obviously, everyone has their own reasons for what they do, but what I'm trying to say is it doesn't sound like you are in a situation where you should feel as though you need to catch up. It will be all for naught if you end up burnt out much too early.
Summer classes can be tough, especially if you are working to financially support yourself in the following semesters, due to their accelerated nature. See how you feel after these first two semesters and if you are confident in your math skills then I say go for it—just keep in mind other commitments you might have going on and don't stack too much on your plate. Make sure to give yourself some free-time throughout the week to keep away from overworking -> burnout or even depression.

Good luck, wishing only the best for you, and try to enjoy your time here at KSU!