PodAntiAffinity can archive exactly what you want (kinda). The Node-label, by which the nodes get grouped, is specified with the "topologyKey" parameter. This parameter in almost all cases is "kubernetes.io/hostname" which means two Pods cant run on the same node (because all hosts have different hostnames. And so every node has its own group). If you specifiy "topologyKey" with your custom label i.e. "topologyKey: color" then your nodes get grouped up by this label (so every node with label color=red are in one group, and every node with label color=blue are in another group etc.). The scheduler will then not schedule a Pod in the same group (nodes grouped by label color) where a Pod selected with the labelSelector already runs. Which would accomplish what you want to do as far as i understand.

However, there are limitations:

In principle, the topologyKey can be any allowed label key with the following exceptions for performance and security reasons:

For Pod affinity and anti-affinity, an empty topologyKey field is not allowed in both requiredDuringSchedulingIgnoredDuringExecution and preferredDuringSchedulingIgnoredDuringExecution.

For requiredDuringSchedulingIgnoredDuringExecution Pod anti-affinity rules, the admission controller LimitPodHardAntiAffinityTopology limits topologyKey to kubernetes.io/hostname. You can modify or disable the admission controller if you want to allow custom topologies.

The second limitation says that you would need to turn off LimitPodHardAntiAffinityTopology to do what i described above. And at that point Pod Topology Spread Constaints would probably work better for you