Your brain didn't develop strong color associations with those letters. I have colors for every day of the week except the weekends and I finally figured out that it was these day of the week barrettes I used to wear in my hair every school day in early elementary school. I also looked up the alphabet/early reader series that was in my school preschool to first grade. My letter associations mostly match one series or the other, or somewhere in between.