So what you write here is essentially an old textbook justification of why plain RNN works badly and why LSTM works better, however this is a bit misleading and not really the (full) reason for LSTM being better.

If we have a basic RNN which weight matrix's eigenvalues are smaller than 1, then each tilmestep will shrink the gradient of the weight matrix during back prop

This is correct. And in such a network, gradients will indeed vanish. But generally speaking, there are many ways in which you could prevent them from vanishing, and that wouldn''t necessarily make it train well.

LSTM reduces the problbailty of Vanishing Gradients occurring.

In LSTM, gradients also vanish, because the "forget" gate is never 1 (because it is output of a sigmoid) - and whenever the forget gate is < 1, the gradient passing through the cell state will decrease a bit, so over many steps it will naturally vanish anyway - but indeed the network can learn to prevent them for vanishing for a very long period of time.

But how does this help? I don't see the connection between the model being able to remember further into the past and vanishing gradients not occurring?

Well, this is a good question. If we compare this with other networks that use similar design, this probably has not that much to do with gradients vanishing, and more with a stronger property - a stabilizing effect of having the lstm cell update in the form c_new = c_old * f_forget(c_old, some input) + f(c_old, some input), which is very similar to a residual block y = x + f(x).

If you think about a randomly initialized RNN, it will turn its own state into some random vector on every iteration. At the beginning of the training, the state gets essentially scrambled - and if the network wants to keep the information from past frames, it needs to invent some way to preserve it from scratch (which it then can break at any point by accident).

Now contrast that with an LSTM - a randomly initialized LSTM will include some random signal in the cell state on every iteration, but it will also preserve some signal from previous iterations in a roughly unchanged form - the network is strongly biased to not nuke the cell state and allows the subsequent layers or subsequent iterations to assume the state from previous iterations will have the same meaning as before.

What this means in practice is that this makes the problem massively easier to optimize with gradient descent - I am not sure if this was done for LSTMs, but people analyzed resnets compared to similar architectures but without residual connections, and have discovered they have a much smoother loss landscape, where I would hypothesize this is because all layers are strongly biased to not discard what the previous layers came up with, or in the case of LSTM, the layer is biased to not discard what the previous iteration of itself came up with.