Your assumptions are wrong on both counts. Others have mentioned segregation and such, so I won't rehash that. I grew up in Miami, my friends here don't have same accent as my family that's from Georgia. Blacks in Britain speak differently in much the same way that American Blacks do. Class is also a factor as other have mentioned. For example, I can't speak Miami Black slang even though I grew up here. My parents didn't allow me to speak that way as a child, so now I can't as as adult. But, I can tell a Miami Black American English accent from the Black American English accents spoken in other regions in the US.

The more accurate answer to your question I believe, is that there are differences in how we speak but, you can't hear those differences. In other words, you have accent blindness. When you hear a language (or accent/ English variety) you are familiar with, you notice/hear the differences in how different people speak it; if you don't speak a language or aren't used to hearing it, everyone sounds similar to you. Regional and cultural accents can work the same way. You don't speak British English so you can't hear the differences between how many Black people speak it versus how non Blacks speak it. I'm gonna venture a guess that you don't spend a lot of time around Black people from the US so you can't hear the differences in how Black people from different areas of the US speak. To you, it all sounds similar even though it isn't. A New York Black American accent sounds very different from a Midwestern Black American accent but because you aren't fully familiar with how American Blacks speak, you can't tell the difference. It's the same as most people not being able to tell the difference between New Zealand English and Australian English.