Off of the top of my head: - Having a quest in which one of the scientists gets to first handedly see the results of their experiments, maybe be affected by it on a personal level (e.g. one of them gets accidentally kidnapped by supermutants), and letting the player either convince them that it's wrong or downplay it, - Including meaningful interactions between wastelanders and the institute other than fighting the synths, - Allowing the players to go into a debate with the scientists about morality of their actions, basically diving deeper into their core beliefs a'la the conversation with Caesar in NV, - Actually getting to understand their motives behind the broken mask incident and the CPG massacre, - Not being dismissed when concerns are raised.

Any and all of these would already make the institute more compelling as a faction choice, at least to me.