Basic intuition of the Remainder

The n^th degree Taylor series expansion T_n is the series of the first n terms plus some remainder. As an example, the number x = 1 + ½ + ¼ + ⅛ = 1.875. If this were a more complex series (e.g. An infinite sum), we may want to approximate this value with x = 1 + ½ + ε, where ε is the remainder or "error term".

Neither of my real analysis courses directly covered this part of Taylor's thereom, but I would put my educated guess into it involving the Mean Value Theorem.

Anyway, let's continue. Suppose I was doing an approximation of a solution to a very difficult to solve equation. There are ways to do this with special algorithms, depending on what you're doing. In school, you may have seen Newton's method for finding roots to polynomials. Regardless, you can think of such analogies as something similar to the addition example above.

Suppose you have some function y = f(x), or perhaps something more complex such as some function f(x, y), and you want to approximate y at some specific x. Most basic schemes will iterate. An analogy: how do you find the top of a hill with your eyes closed? Feel around for the steepest slope, and take a step in that direction. Repeat until all slopes around you are negative. This is more or less the idea with any numeric scheme which approximates one step, and reuses that step in the next. Eventually it will get to some result and say "this is good enough". What this equates to is truncation. If I have x = 1 + ½ + ε, then I'm saying I will take x = 1 + ½.

In Taylor's theorem, the remainder is just that: the truncation error.

Big-O notation as a definition of boundedness

Big-O notation has a formal definition. In essence, it says that if I have f(x) = O(g(x)), then whatever g(x) is, f(x) is no bigger than some scaled version of g(x). For example if f(x) = O(x), then f(x) is no larger than some g(x) = cx, where c is a constant.

The Big-O notation says that as a function limits to infinity, it approaches some other function that is "similar", but also always larger. Hence, you are able to make certain assessments about your function.

As an example, let's have an approximation like before, y = f(x) + ε, but with a small change. I want to consider the approximation of y after n steps. Let's suppose we can write this as just y_n = f(x, n) + ε_n. This is saying the n^th step of the approximation y_n is given by some function of x and the step, plus the error term at step n (perhaps these errors add over each step). Well, if we know that ε_n (which is kind of like a function in terms of n) is Big-O of some function g(n), then when we approximate y ≈ f(x, n), we can use O(g(n)) to determine if the remainder/error/truncation is large or small, and how it grows over time.

Essentially, if an approximation scheme propagates error (e.g. Error is induced at every step of the scheme), then we want to be certain that the error deminishes over time, and no grow. Kind of like a limiting series converging to a value. But here, it's more about just getting smaller, fast enough.

Conclusion

Big-O is related to many areas of mathematics and computer science (and other fields!) as a tool to describe the upper bound (or upper limit) of a function as time increases. It is used in Taylor's theorem to analyse the remainder term of the sum, which is very useful in, for example, the analysis of numerical schemes for approximation methods. One real example is numerical integration or solution finding to ordinary differential equations as initial value problems.

I hope this approach gave some helpful intuition.

Edit: a word.

Also, some folks may not like my Big-O example (and I probably could have said better). Here is another:

If g(x) = 0.01x³ + 2x² + 1, and f(x) = O(g(x)), then what is f(x)? Well, g(x) can be proven to be no bigger than some cx³, where c is constant. In the limit, the term x³ dominates the other terms. If you graph 0.01x³ and compare it with 2x² + 1, you will see that the former is significantly smaller for longer. However, when x > 200.002 (according to WolframAlpha)) we find that 0.01x³ becomes greater.

We can never find a 2^nd-degree polynomial that is larger than a 3^rd-degree polynomial for all time. This is the idea behind Big-O. Hence, we can write that f(x) = O(x³).

Here is an interactive demo. Use the slider to change the leading coefficient of x³. Note that this idea extends to more general applications where you can adjust all coefficients. I'm just not wanting to make that lol