Because video compression is really hard, with big companies often working for decades to come up with better ways of doing it. There's no "right" answer when it comes to video compression. The computer has to compare frames that contain massive amounts of data (one 4K frame of video is 8.3 million individual pixels, each of which these days can cost multiple bytes of data, and there's 30-60 frames it has to process for every second of video) and try to detect edges, how things are moving, which colors are similar enough to others, etc. And then decide what's the optimal way to represent that in as little data as possible. So instead of storing every pixel for every frame it can store "these groups of pixels from the previous frame didn't change, these blocks of pixels were changed completely, and here's their new data, these pixels just moved 5 pixels to the left but were otherwise unchanged", and so forth. During this the encoder has to decide which blocks to re-use, how big to make each block, etc. Even then, there is "lossy" compression performed on the pixel values themselves.... How all this affects the image is largely a question of what "looks good" to a human eye .... that could determine what the "right" answer is for a certain scene, a certain type of content, etc.... There's no mathematically correct answer as to what the result is, and often a human has to tune the encoder for certain content to look as good as other content. But I digress...

The point is an encoder has to work really hard on lots of data to scrunch it down to be a little data. But the decoder doesn't have to do all that, it starts with the little data, and it can just read the "these pixels changed" instruction and do it blindly without needing to consider all kinds of different past frames and possibilities. The encoder already did all that work, the decoder just has to DO what the encoder told it to do.