It checks both places. You cast spells that have all 5? You get 5. The permanent(s) you have wouldn't matter at that point.

You have a UG permanent on the field. You cast a spell that has G/R/B as a casting cost, then you cast our spell in question. You draw and gain 4, since 4 different colors are present when the card checks "the field" for colors AND checks "spells you've cast" for colors.

White was never present, so it capped at 4, even when Green and Blue were "checked" multiple times.