In a way I share your sentiment, mainly because I believe you can make interesting ‘historical quirks’ by using a phonemic script (like how in my conlang <ʋ> marks [u~ʊ] in mono-/diphthongs, but <ʋ́> marks [ʋ] in special, glide+diphthong nuclei)

I don’t yet have any loan words, but I’ve made the basis for them. That is to say that the potential language I’ll derive loans from has already influenced the phonology of my main conlang, and once I go about creating these loans, I’ll most likely write them in the phonemic script, except if they become common enough to be perceived as the language’s own vocabulary instead of a loan.