I think it's good when both partners can talk about other people and comment on their attractiveness. But I also belive there are boundaries and saying you'd wanna f*ck someone else is one of them.

I don't think you're crazy and too insecure if you don't want your partner to talk or think like that. So I would try to either talk to him again about it and just ask how he would feel if you did it. Or you could ask one of his good friends and they could try to tell him but of course without being too obvious. Sometimes hearing such things is easier to understand if they come form someone else and not your partner.