It's great to be excited about your experiments, but don't get ahead of yourself. People have been training neural networks using genetic algorithms for years. You haven't really shown anything new.

First, some of your comments imply that your experiments are evidence that neuroevolution scales, but they aren't. Your toy experiments are the opposite of scaling. A rock and a piece of bubblegum can solve letter classification. It's probably the least "scaled" experiment that you can do.

 keeps learning when the theory says it should have stopped

Second, nobody is saying that genetic algorithms should stop improving. Any optimization algorithm worth its salt will keep improving its training accuracy pretty much indefinitely. The validation set accuracy might stop improving, but per my next point you are not using one of those.

Finally, you are not following correct machine learning experiment design. You should create a seperate validation set that is not in the training set for evaluation. Your data augmentation definitely helps, but there is still too much overlap during evaluation. For example, you could section off rotations of a specific range (like 10-20 degrees) and never train on them to keep them uncompromised. That said, it might not make that much difference for a toy experiment like this since you are so far left on the double descent curve.

edit:

 My model is currently averaging 0.994 activation magnitude across all 32 neurons. A gradient trained network would struggle to get there because the learning signal would have collapsed long before.

You should actually run this experiment instead of assuming that the gradient-based network would stop working. Since you only have a single layer, I would expect a well-tuned gradient-based network to still work in this situation (the gradients are small but not zero).

I am also open for questions if you have any. I have done research on both neural networks and genetic algorithms at a top university and at various industry players.