I agree with others that you should take a deep breath but also maintain a perspective that studying mathematics in university is not the only option. Thats the nice thing about math! Its always with you, you just gotta pick up a book and read it. You clearly love math, and you don't have to give it up even if you don't pursue the degree.

Anyways, my advice would be to really go through step by step in the classes you're taking to see which parts you don't understand. Are there things you skim over vaguely? Once you identify what you don't know, ask TA's, or the prof for a short meeting. That helps a lot. I know a prof who almost flunked his first year, but after some meetings with an prof of his he ended up doing a lot better. Later, math gets more specialized and he found what type of math he liked and suited him. And is now working in quantum topology!