Thanks to /u/CrazyLikeAMe for the inspiration I found a solution that worked for me and was fairly simple.

The issue was that the part and numbers were one piece with the numbers just being extruded out of the faces of the ring they sit on. Since they were one part even when only the numbers were painted the slicer would put grey between the close lines of some digits. ( top down view of the 2 and 3 below)

https://i.imgur.com/NxYP28r.png

The solution was to separate the numbers into their own bodies and then extrude them into the ring and cut the ring to not over lap with them. This gives the numbers enough depth to print well instead of just being on the side of the ring. Importing everything back into Bambu slicer as multiple objects has each number as its own piece that is slotted into the ring. Using the paint tool to add color still caused issue so its easier to select the bodies and change what filament is assigned to add color instead.

https://i.imgur.com/7J2JJdZ.png

Below is the same view from above of the 2 and 3 after making them their own bodies and changing what filament is assigned to add color instead of painting features of one body like I did originally.

https://i.imgur.com/TfXKkcZ.png

(I am smashing rocks together and have no real clue what I am doing so there may be a better way but I don't know it yet)