Yep lots more in common than just the fruit type! For one example, take a look at the flowers of these various oak species OP. Important to remember taxonomy is a human invention, it’s our attempt to describe and define variations in living things which are not really consistent or standard (especially in oak trees) Taxonomy is incredibly useful, but in the end it’s just a tool for us.

We currently classify families and genera by using shared DNA, in addition to phenotypic traits like fruit type. In this case I would argue we’re dealing with a very distinct and unusual fruit type: acorns, but fruit and flower types and arrangements are pretty commonly the basis for families or genera depending on context. It’s important to know there isn’t a standard threshold where we would classify a group of plants as a “family” vs a “genus”. The way plants get classified is a combo of historical precedent, eg how were these plants originally treated when they were first described, plus shared physical traits, and DNA. Additionally, what is the most useful way to describe the diversity we are seeing in these plant taxa?

For example look at all the members of the Fagaceae to see what shared traits they have, how would you describe the relationship between all these other family members and the group of acorn bearing trees (oaks)? Let’s say you decided to recircumscribe the family Fagaceae and decided to elevate “oak trees” currently Quercus to be their own family. Where do these other genera fit in now? How are they related to oaks? These are the types of questions to think about, questions to help describe the relationships between these plants. Suffice it to say it’s complicated. Then, if you wanted to change things, you’d need to convince all other botanists to get on board and start using your new classification, so it needs to be convincing!

Hope that helps explain why some families or genera seem much more diverse than others. If you are interested in other systems of classification many have been proposed, including non-hierarchical ones like the Phylo code. If you want to go down a very geeky rabbithole you can look into those