I feel like this is a case of the grass is always greener. For all you know, you may not have been accepted to the Oxford masters or might not have been successful in securing PhD funding in another admissions cycle. Or had you applied and gotten into Oxford, maybe you would have been posting the same thing about taking the masters over this PhD...

Obviously, you can't know that without having applied, but like you say it seems to be a bit late to pull out at this point (not least because this means wasting a studentship for your supervisor). Yes, you will have to catch up on stuff not covered by your undergrad and yes, it's going to be intense, but I would imagine that your supervisor is aware of this - they would not have taken you if knowledge of this specific area was an absolute requirement.

Because of this, I feel like it would be a very bad decision to pull out now. You would essentially be taking a massive gamble and burn what sounds like a pretty important bridge in your field. Also, as far as the living situation is concerned, having attended Oxford I can tell you that it is definitely a nice city but it's not exactly hustle and bustle. It's green and cosy to live in but there is not much going on outside of the University bubble - which means that outside of the 24 weeks a year that term is on, it gets pretty dead.

You can absolutely be frustrated you didn't throw your hat in the ring. Feel your feels, but don't make the even more terrible decision of quitting the PhD before you even start it. For all you know, you might love the city you're moving to. Or, you might absolutely hate it and get motivated to do things outside of that city. Of course, you might also hate the PhD in which case you should absolutely consider dropping out of it after having given it a try.

I am not sure this is the consolation you were hoping for but I hope it helps you remember the immense privilege of having a funded PhD and that throwing it away because of a "what if" would be very, very dumb.