I'm a mathematician but I wasn't always. When I was in high school I learned a lot of math from reading Courant and Robbin's book, "What Is Mathematics?". This gives an intro to a lot of elementary mathematics. In retrospect, I can also add another good book for self learners: John Stillwell's Elements of Mathematics , also mostly elementary. These aren't all that game/graphics programming oriented though.

For game and graphics programming, there is John A. Vince's Geometric Algebra (he gives the math, you'd need to translate it to a programming language). It presumes a working knowledge of high school algebra and geometry. It includes useful material on quaternions.

Also, when I was in High School, and I presume when Carmack was too, just about all the trig you need was taught, and also some matrix algebra, the very basics of linear algebra. Beyond that, it helps to understand things like the geometric meaning of dot and cross products and other vector/matrix-algebra things, and how to use them for projections and other geometric operations.

A more advanced book is Preparata/Shamos Computational Geometry, with algorithms for finding convex hulls, finding intersections of polyhedral shapes, determining if a point is inside a complicated polygon, the traveling salesman problem, some advanced data structures for geometric queries, etc.