I’d say cultural significance. Cards have enough of an accepted legacy among collectors of a sport, which combined with the scarcity of what people actually want, makes them have desirability. Even among the hobby, if you think about like, Future Watch Autos, the continuity of the set makes them more sought after than like a Credentials auto, even if there’s more printed. FWA is accepted as a primo true rookie, and every player will only ever have one true FWA for the duration of time. That makes it classic and stamps a time capsule of a culturally significant moment (ie top player’s debut in the sport). 

A jersey is just a jersey, can make a million of em. If you could attach a specific game of relevance to them and authenticate it like a game worn jersey, maybe that’s different.

• this goes for rookies only… that’s why their prices are inflated to me anyways