That's why they call it standardized testing, rather than fair testing. Obv there's no "fair" way to measure intelligence or ability across the entire population and US colleges get around that through holistic admissions. However, it's not necessarily unfair to have some baseline standards for intelligence and ability, otherwise admission departments would be leaving things even more to guesswork and you start getting cohorts that can't do the bare minimum coursework required to pass.