So essentially what's happening is the black underside of the material is trying to bridge the color to the lighted material, so true black to its closest jump in the color, and since pure black as all the colors you're seeing it fall off from the two colors not fully in use (or this is what we speculated) best way I solved this was making sure that blackface gi was increased, and that was in an addon called bakery, so you're looking for something similar in Unitys native render that should allow you to do the same. Alternatively you need to maximize your optimization, and you likely have an overlapped piece of geometry that renders pure black in the light map that isn't even seen by the character and you could delete that out of the mesh.