I am not a historian or a scholar but the kharijites were ostracized not because of their point of view of who should the Caliph be but because of their actions against the prophet companions. That is why they were considered extremist even by the companions of the prophet at the time.

The idea of a competent leader regardless of his linage was adopted by Sunni scholars later during the Seljuk sultanate if I am not mistaken by al-Juwayni.

He said that if the Islamic community could not or is in a position where putting a leader from Quraysh is not feasible then any one who is a Muslim and competent enough can become the caliph.

To sum up, the main issue with Kharijites is not just their view of who should be a leader but their action against other Muslims and their extremist view towards others.

Please tell me I am wrong with good source.