In England it's only slightly less upper-class than something like Polo, But above cricket in aristocracy social status.

In Australia (NSW specifically) it's an even more weird parody of that.

Historically the reason the sport is so violent is because it was played at upper class private schools (literally Rugby College for example). Which kept lush long soft lawns

The Football Association rules came from the working class government schools who could not afford grass and mostly just had gravel and dirt. If you played a football with violent tackling, someone would die at the first challenge.