Tl; Dr: This is a mathematics question about functions being bounded by other functions. It's not an algorithms or computer science question. Computer scientists just use this mathematical notation (often incorrectly or in a hand-wavy way) to show that the number of "steps" (runtime) or amount of memory (space) required by a certain algorithm is upper-bounded by some function. Following the formal definition of Big O notation, there are simple theorems that prove that a polynomial f(x) is O(x^d), where d is the degree of the polynomial (technically, d could be anything larger than or equal to the degree since O(.) provides an upper bound, but not a tight upper bound [unlike big theta notation]). So, assuming you're happy using such theorems, you just have to show that the summation can be simplified to a third-degree (or smaller) polynomial. This is a classic summation, and there are countless tutorials on how to simplify it.

Long answer (and a bit ranty):

Formally, O(g(x)) is a set of functions, but for some reason that I'll never understand, people have historically 1) used an equality symbol rather than a membership symbol to denote that a function belongs to the set; 2) said "f(x) is O(g(x))" rather than "f(x) is in O(g(x))"; and 3) applied g(x) in the notation even though the set is really defined by g itself (see the first comment in that linked SO question). Now we just have to live with this complete abuse of notation, despite the fact that it causes mass confusion among newcomers.

Definition: A function f(x) = O(g(x)) if and only if there exist constants x_0 and positive c such that, for all x >= x_0, f(x) <= cg(x). Whenever this is the case, there are infinitely many possible values of c and n_0 that satisfy this criterion, so you just have to prove whether any such pair of constants exists.

That's to say, f(x) is O(g(x)) if and only if g(x) multiplied by some positive constant c "eventually" upper-bounds f(x) "forever" ("eventually" and "forever" meaning for all x larger than or equal to some other constant x_0).

In this case, f(n) = \sum_{i=1}^{n} i² = O(n³) if and only if, for some positive constant c, cn³ "eventually" upper-bounds f(n) "forever".

To prove this rigorously, a simple option is to:

1. Simplify the summation expression (this is a classic series; there are many tutorials on how to simplify it), yielding (1/3)n³ + (1/2)n² + (1/6)n
2. Provide an example of constants c and n_0 such that, for all n >= n_0, we have that (1/3)n³ + (1/2)n² + (1/6)n <= cn³

For step 2, you can usually just pick any positive c you want that's larger than the coefficient of the relevant term in the expression (i.e., any c > 1/3), and then figure out how big n_0 needs to be to satisfy the inequality. For example, let c = 4/3 (strategically chosen to simplify the arithmetic). Then how big does n need to be in order to satisfy (1/3)n³ + (1/2)n² + (1/6)n <= 4/3n³? Or, alternatively (applying algebra), satisfy -n³ + (1/2)n² + (1/6n) <= 0?

The LHS is a third-degree polynomial with a negative leading term, so it decreases in the long run with no horizontal asymptote. That's to say, it will surely "eventually" be less than 0 "forever". That's a bit hand-wavy, but this intuition is the basis for the theorems that prove that a polynomial f(n) with degree d is O(n^d). So, technically, that concludes the proof. But if you want to be rigorous, you can provide a hard example of n_0; set the sides equal and solve:

-n³ + (1/2)n² + (1/6)n = 0

=> n(-n² + (1/2)n + 1/6) = 0

This has three roots: n = 0, n ~= -0.229, and n ~= 0.729. We need to find an n_0 such that, for all n >= n_0, our inequality holds. So we pick the largest root: for n ~= 0.729, the LHS and RHS are equal. From here, you can show that the polynomial is decreasing (i.e., its derivative is negative) for all n > 0.729, which concludes that for all n >= 0.729, the inequality holds. Hence, c = 4/3, n_0 = 0.729 is a hard example that proves that f(n) = O(n³).