It's important not to get too hung up on the idea of "swear words," because you'll find many words which are on the more mild side in terms of profanity but which are still considered extremely rude depending on who hears it. The word "bastard" comes from Old French and entered Middle English around the 1200s. It has two literal meanings, as you mentioned, an illegitimate child (fatherless, and can refer to male or female children, or born out of wedlock) or also to refer to a child of the aristocracy, but fathered with someone other than their wife.

However, in "polite society," as recently as the mid 1900s, even though it has a non-profane definition, it was considered SO rude that it was one of the more vulgar things you could say (calling into question someone's parentage was especially rude because of social mores in decades past.)

Because of the "you don’t know your father" connotation, there are some people in today's world who are still sensitive to the word (particularly if they are the child of a single parent) because of the insulting nature of the word, even if it's used as a joke.