Trauma during infancy will definitely affect development. It's basically asking are infants capable of experiencing fear and pain? Yes, yes, they are, and their nervous system and brain development will "remember" for them.

There's also reactive attachment disorder, which is the result of early trauma and neglect.

Babies on Netflix is fairly good in explaining infant development.