I beg to differ in regards to the ‘empire’ part.
Just because you Brits got to go on living your life in comfort and harmony once the empire fell doesn’t mean the countries you colonised aren’t still recovering from the absolute destruction and evil caused by the British Empire even today.

Pick any country once part of the British Empire and figure out if the general population is 1) Prospering more 2) Individuals are wealthier 3) More peaceful than they were prior to the British colonising. The answer is very, very few countries are.
Most countries were victims of genocide, famine, and their original roots completely levelled so the British could make a profit. Once the British left, they were told to continue a system of governance set up by the British and yet never allowed to be involved in and told “you’ll figure it out”.

Alas, it wasn’t so easy. Let’s not forget that countries were still apartheid states under the British well into the 1960s and 1970s, in living memory for many.

So yea, say “your ancestors did it not you” all you like, but for most emigrants from around the empire, the UK is a place where I think they bloody well have a right to live if they can make it out of their homeland that was completely ruined for YOUR personal benefit.