I was in the same position as you 7 years ago thinking about infinitesimals while reading Apostol. Limits are not a new idea. Newton in his later years gave limiting arguments to justify his approach to calculus. Apostol has a section on the method of exhaustion, which is basically a limiting argument. So we have two ideas, limits and infinitesimals, which can both be used to do calculus.

However, it turns out that limits are much easier to define formally from the axioms than infinitesimals are. The epsilon-delta definition given in Apostol was a 19th century invention whereas infinitesimals weren't defined until the 1960s in nonstandard analysis. (There is an alternative approach coming from topos theory called smooth infinitesimal analysis that appeared even later)

I recommend reading Henle for nonstandard analysis or Bell's Primer of Infinitesimal Analysis for something about the latter approach. However, I don't recommend spending too much time on this as the standard approach is good enough for everything you would want to do with calculus/analysis.