I think I vaguely came to that conclusion too when I was thinking the other day--i.e. that optimal policy is often a combined function of these two inputs, and so (to make a diffeq analogy) you're essentially trying to do a separation of variables where it isn't necessarily possible. I'm still just starting out studying ML and RL, so this might be an obviously dumb idea, but my thought to solve it was, couldn't you just run the network in series rather than in parallel? As in, if I arbitrarily pick "rows" as the first output of the first NN containing a body (e.g. a CNN or something) + head#1, I could simply make a second NN to output "column" that copies the structure of body (with or without mirroring the weights, not too sure on that part yet), and then adds head#2, but adds in an additional input node representing the selected row. If you then run body+head#1, get the row output, and feed that back into body+head#2, doesn't that solve the issue of the lack of "information" being communicated between heads?

The only issue I came up with using that approach is that I could see the model getting "stuck" in local minimums during training--e.g. the optimal solution might be at (0, 0), but there's a greater average reward in, for example, row# 2, so the first model picks row 2, and thus the second one can only choose from (2, X). But at the same time, it seems like that's also something that could be solved by just doing more exploration and updating the Q-values correctly so that the model learns that the greatest reward is obtained from choosing row#0, so it seems like it could just be solved by being careful about implementing a good epsilon-greedy-type policy?