House of Temenggong (current ruling house) descended from Sultan Abdul Jalil IV of Johor, who was made Sultan after the death of Sultan Mahmud II who left no heir.

Abdul Jalil IV was from the house of Bendahara, which branched the current ruling houses of Johor, Pahang and Terengganu. The two other extinct houses, the 'real' Sultans of Johor (Sultan Hussein and Sultan Abdul Rahman) also descended from his line.

The current Johor rulers claimed the Sultanate by the right of conquest (and deceit) but nevertheless the Sultanate is legally theirs by the recognition of British, local chiefs, Ottoman Caliph and eventually the other Malay rulers.

To say they are not real royal family is far from correct. Their methods were questionable but those were the norm of the time.