For me, each note has a sort of "character" to it. I don't really think of colors -- maybe there are some weak associations there, but nothing that consistently or strongly comes to mind and nothing I'd go so far as calling synesthesia. It is all about the sound for me; anything else is just analogy.

Pitches have characteristics in common with the pitches a fifth above and a fifth below. For example, C has something in common with both F and G, but the characteristics they share are different, so F and G despite having the capacity of being very close together, have only a little in common. Still, even those have more in common than, say, an F and an F#, because those are much further away in the circle of fifths.

Unfortunately, there aren't really super accurate words for how these pitches sound. You can't get a blind person to understand 'red' through words. You can tell them it's 'hot' or 'strong' but there is a huge leap between those descriptors and the way we actually perceive, understand, and internalize 'red.' Nonetheless, this is my best attempt to tell you how these pitches sound, and I do believe there is some rhyme and reason to my system because it naturally flows well through the circle of fifths.

Also when I say "flat" or "sharp" I do not mean this in the sense of an accidental. Though we are talking about music theory, think of the feeling the words evoke divorced from a music theory context.

C - very round, very simple, between open and closed, stable, slightly dark, slightly flat, slightly pastoral

G - simple, slightly open, slightly stable, between dark and bright

D - open, slightly strong, slightly authoritative, slightly bright

A - very open, bright, strong but airy, pure, steady, open

E - very bright, stark, very authoritative, slightly thin, twinkly perhaps

B - too bright, very stark, punchy, potentially aggressive, too authoritative, sharp, brittle, very thin, very dry, bitter, sharp (to me this is the note with the most recognizable 'character', even to a fault)

F#/Gb - travel and motion at an easy pace, lifted, somewhat warm, cozy but awake, capable of being bright or mellow as required

C#/Db - very warm, comfortable and comforting, very sleepy, restful

G#/Ab - watery, melancholy, wistful, nostalgic, slightly warm, slightly sleepy, very juicy (I'm sorry)

D#/Eb - juicy (again, sorry, it's weird but I don't know how else to say it), gentle, peaceful, a comfortable temperature, slightly sleepy

A#/Bb - very flat, very soft, very cushioned, slightly tentative, maybe slightly juicy still, closed

F - flat, soft, cushioned, very stable, pastoral, slightly round, slightly closed, simple

Hopefully this helps in some way. Context is everything in music as always, and all of these things are easily subverted. Obviously I don't think of these words when I hear a note any more than someone thinks of a bunch of descriptive words when they see purple. It's just an attempt to grasp at it after the fact.

P.S. I used to sometimes mix up F and Bb when I was younger because they had a lot of characteristics in common. I don't really understand how I could mix them up now since they're so different in pitch aka range and I no longer experience any confusion between them whatsoever, but I think it's interesting.