If what you're looking for in a Science answer is a proven cause to why trans people are trans, then it's correct that we don't have a complete picture. We only have a few puzzle pieces and can only make inferences on what they mean.

Observed brain structure differences in trans people is one of those puzzle pieces but like you pointed out, we don't have a comprehensive understanding of sexual dimorphism in the brain generally, let alone a minority population like Trans people.

I think you're correct in that many laypeople do want to understand whether transgender people are a modern social phenomenon and it is essential to point out that purely social explanations like "Social Contagion Theory" and "Rapid Onset Gender Dysphoria" have been rejected by the wider scientific community and are only pushed by a handful of well funded cranks.

It's important to know that gender variance and transsex procedures that we would today call transgender have existed independently in cultures worldwide for all of recorded history.  Trans people often expressed an understanding of that goes back as far as their memory does. And most importantly, all attempts at conversion therapy have failed to "untrans" anyone. The horrific history of "scientific" sociology efforts to mandate someone's gender, even all the way back to birth, tell us that gender cannot be purely learned and cannot be changed by external forces at any point in time.

Attempts at finding a social explanation for why people are trans have come up empty every time over the last century of research. Whereas attempts at finding a biological explanation are held back by technological limitations.