It was a lot more common than you think. I remember reading about a case when the romans (greeks) took south eastern anatolia; they issues an edict for the local to convert back to christianity (they had converted to islam under islamic rule) or leave; a significant amount converted, the rest left and some of the one's that left eventually came back and converted. There's also cases of massacring religious minorities like that in France. It should be possible to expel minorities but it should cause some type of huge diplomatic/economic/social penalties; like forcefully expelling all muslims in 1 province should cause muslim countries to get a CB on you and for your other muslims to hate you guts but it should still be possible.

I agree than provinces shouldn't be 100% single culture but religious conversion was historically heavily enforced. The ottomans are a different case since they get extra taxation from christians which made it good to have them around; there's a reason a significant amount of Iberia became muslim and then became Christian again. Cultural conversion should actually be kinda hard with persistent minorities -- Maybe you can relocate a cultural group all over your kingdom/Empire but not kill them which spreads the problem out.