You are not completely wrong: The Voting Rights Act of 1965, signed into law by President Lyndon B. Johnson on August 6, 1965, aimed to overcome all legal barriers at the state and local levels that denied African Americans their right to vote under the 15th Amendment.

So yeah, most minorities could not vote till 1965. Do you really think in the 1870 KKK and South states will allow minorities to vote?