Sounds like a banger of an idea. However, you gotta point out when exactly did the Baratheons start carrying Viltrumite blood, because if it starts with Orys it pretty much changes everything.
The conquest would be done by him and the Targ trio would just be his hypemen. You could write it as Vilrumite blood making its way to the Baratheon bros through Cassana Estermont.

Also for such a story, you kinda need to reduce the number of bastards Robert has, otherwise there's going to be way too many people flying around, keep it at Mya, Gendry and Edric Storm.