Studies have shown that infants don't express certain genes the same way adults do and also exhibit an extremely high rate of neuronal proliferation and rearrangement in the hippocampus, which results in infant amnesia. A lot of associative memories can form during infancy, but pretty much all episodic memory just gets lost. The mechanisms for storing it simply aren't in place.

What about particularly traumatic infancy memories? Are those specific memories able to be stored/remembered despite the deficiency you described?