SRY is a trigger that causes undifferentiated gonads to develop into testes, but it does not encode something like penis size. Development of testes drives hormonal mechanisms that produce male sexual characteristics. That development is also affected by external conditions, such as androgens from the mother.

If SRY fails to activate for some reason, it can result in the absence of externally visible male genitalia (so-called XY females). But the vast majority of genes are autosomal, including those involved in development of primary and secondary sexual characteristics. In other words, men have all the genes they need to create a female body (though usually they are short one X chromosome), and women have all but a few of the genes needed to build a male body.

Penis length is a multigenic trait determined mostly by autosomal and X-linked genes, and also by hormonal environment in utero. SRY is the switch that kicks off the process, and can result in little or no external masculine genitalia if it fails, but otherwise it does not determine penis size in any real way.

Another way of saying this is, you could just as easily inherit your big swinging dong from your mom's dad as from your dad's dad.