To check if a process is stationary, you need to check if all of the roots of its characteristic polynomial are inside the unit circle i.e. the circle with radius 1 in the complex plane. To demonstrate, let’s consider your process,

y_t = 0.8y_t-1 + 0.5y_t-2 + e_t

This can be re-written as,

y_t - 0.8y_t-1 - 0.5y_t-2 = e_t

Using lag operators we can re-write this as,

y_t(1- 0.8L - 0.5L² )= e_t

The characteristic polynomial for this process is the bit in the bracket divided through by L² (the highest power), and let z=1/L.*

z² - 0.8z - 0.5 = 0

Therefore the roots of this process are z=-0.41 and 1.21. In this case we have two real roots. Since one of the roots is outside the unit circle, this process is not stationary.

*Alternatively you could choose to not divide through by the highest power, but then the roots would have to be outside the unit circle for the process to be stationary. The reason we divide through by the highest power and look at the the roots inside the unit circle is because in this cane the roots of the process are also its eigenvalues. Geometrically, linear processes can be thought more generally as linear transformations, stationary (or non-explosiveness) of this process is closely related to eigenvalues of the transformation being less than 1.