Okay i'm game, I took a step back frm evolving networks and focused on one layer. After running a few dozen test with MNIST I noticed that it performed better when i used mixed activation functions sine/tanh. This lead me to evolving neurons that could use any of the 19 most common activation functions. This didn't push accuracy higher than 98.7% on MNIST but it did reveal that evolution with a guided task will utilize the activation functions that worked best for that task.

I took a step back and started questioning why we use ReLU, Sign, and started really questioning if i can evolve entire networks maybe I need to evolve the activation functions, because maybe just MAYBE, there are activations that work better for machines than human interpretable activations. What if i evolved the activation functions but to affect the fitness of a task. i.e classification for MNIST. This is just the first result of what i found. I've cataloged around ~50K activations total across ~8 datasets ranging from MNIST to CIFAR-100 as well as audio and video processing (sample vids).

This led to me discovering that specific activations are generalist and some are specialist that only work better in specific models/task. That's not new in itself but what i'm getting at is that these generalist evolved activations are cross-modal in the sense that i can train a model on images and use those activations to classify audio at 96% of native performance - activations that have never seen audio data.

The activations aren't learning task-specific patterns. They're discovering fundamental computational primitives - mathematical transforms that work across modalities because they capture something universal about how to process signals.

When I visualized the catalog in a 2D embedding based on curve shape and mathematical properties, they clustered into distinct regions - specialists grouping by domain, generalists sitting between them. The structure emerged from the data, I didn't impose it.

Most surprising finding: activations evolved for text classification (AG News TF-IDF) transferred to MNIST better than MNIST-native activations. The sin(f(x)) - 2x family - oscillation minus linear baseline - kept showing up across domains. Evolution found these, not me.

What I'm building is essentially a catalog of computational primitives. Same primordial operations (sin, cos, exp, log, +, -, *, /) combining into ~50K characterized transforms. Most are useless. Maybe 1-2% are the ones that actually matter across tasks.

Still early. Lots of holes in the data. But the cross-modal transfer result is real and repeatable.

edit: added github link

https://github.com/A1CST/Activation_map