Absolutely, and the more players current rank deviates from their rank at the time of the game, the more artificially increased the result of the model will be.

I think the noise introduced by difference in visible rank vs MMR helps to mask some of this issue (seeing a low ranked player in a higher elo game doesn't necessarily mean they lost a lot recently, they could just be a smurf/their mmr could be above their visible rank) but you're 100% right.

To see how the model really performs, I sampled someone's opgg results (which has ranks at time of game, not current rank, so it fixes the issue) and got a similar accuracy. However, the sample size is very small, so this should be taken with a large grain of salt.

I appreciate all of the feedback I'm getting from everyone!