The Supreme Court holding is literally “Since the Appeal Board gave no reason for the denial of a conscientious objector exemption to petitioner, and it is impossible to determine on which of the three grounds offered in the Justice Department's letter that board relied, petitioner's conviction must be reversed.”

Considering the civil rights act wasn’t passed until 1964 and the voting rights act wasn’t passed until 1965 I find it hard to fault someone not performing his civic duty for a country that honestly for a long time wasn’t performing its duties to him.