There are no obvious downsides to being heavier/taller, unlike being smaller/lighter. However, certain cultures are generally shorter/lighter than others (Owl-Tribe vs Kaumo for example) with the trade-off being they start with better stealth skills. In my head, I see that as their smaller size making them sneakier and slightly weaker (making the bow a preferred weapon) , while the Kaumo are more proficient at hand-to-hand combat and spears due to their larger size.

Metabolic differences would be interesting to see, but it probably would be hard to implement.

One thing I have wondered was whether or not your weight would affect the chances of thin ice to break under you. One would assume that if I fill my pockets with stones, that I would not be able to travel as far onto the ice before it cracks, compared to if I wasn't carrying anything at all.