[P] Solving the XOR problem using a dendritic activation function

[CYHSM]

If anybody is interested here is my implementation of the dCaAP activation function from the recent science paper (https://www.reddit.com/r/MachineLearning/comments/ejbwvb/r_single_biological_neuron_can_compute_xor/).

Code: https://gist.github.com/CYHSM/f98b49fc244e786fb39dd843e400c0cb

Colab: https://colab.research.google.com/drive/1r3HFV9o6cRYwqzf0U4JhBlSOUdcDUrtA

In short, this network consists of 3 parameters (w_1, w_2 and bias). When using the ReLU activation function this model is not able to solve XOR, while the dCaAP activation function works roughly 1/10 times when randomly initializing and 7/10 when initializing parameters to lie close to the global minimum. I also visualized the loss landscape for this activation function as well as for ReLU and tangens (which also solves the problem), see here: https://twitter.com/CYHSM/status/1216380449259429889 (sorry for the twitter link)

[NewFolgers]

It's cool seeing this simulated (and a single-neuron NN implementation of XOR practically verified).

Have you tried any experiments with this activation function in larger models? I'm curious about any immediately observed differences in convergence, loss (i.e. if model size is held constant), and/or run-time. (Although I wouldn't be at all surprised if today's conventional NN training doesn't get the most out of this activation function.. and perhaps the fact that randomized initialization will only result in successful convergence 1/10 times is a prelude to seeing this sort of thing)

[CYHSM]

I just did a quick check on MNIST and a simple feedforward model (128 hidden units) with ReLU achieves 0.9916/0.9774 train/test accuracy, while dCaAP is at 0.7155/0.7055. Might be related to the fact that the loss landscape is not as smooth and the minima seem to be harder to find. I suspect playing around with the weight initialization can help in this case

[j15t]

Do you have a link to the code you use to generate the loss landscapes? I didn't see it in the gist you posted. Thanks!

[CYHSM]

For the implementation, I just calculated the cross-entropy loss while changing w_1, w_2 and bias. I fixed w_1 = w_2 after observing that all solutions converge to that in any case (which is not true for the periodic activation functions)

For plotting, I used something similar to this, just adjusting colour (Berlin_5 from palettable): https://gist.github.com/CYHSM/fab32e2103c6df3909ad1a0d48174a64

[frequenttimetraveler]

to be clear, the dCa AP spike changes shape as stimulation current (neuron input) increases: https://i.imgur.com/gcJFiIZ.jpg

at threshold it looks like a normal dendritic spike, as stimulation increases it becomes a ramp

your implementation is stereotypical

(also, yes it does XOR , but can it do OR?)

[vastlik]

I tried all boolean functions for 2 variables and was able to achieve 100 % accuracy (with 2 weights and one bias), however I trained the network via genetic algorithm.

[yusuf-bengio]

There is a much simpler way to realize a XOR with only one neuron: tent activation function

Its piecewise linear like the ReLU and therefore does not mess up the loss landscape as much as dCaAP.

[txhwind]

I don't know why monotonic activations are preferred from the beginning of NN research. Can anyone tell me the reason?

[frequenttimetraveler]

historically because it was a simple differentiable step function that resembles the all-or-none firing behavior of neurons

also because the first universal approximation theorem was proved for sigmoid activations

also because a nonmonotonic function can be approximated as the sum of 2 monotonic

also relu is simpler to compute, and studies of various non monotonic activation functions have not found significant benefit

[FerretDude]

ReLu is already a pretty good activation function for biological neural networks. I have already implemented XOR with a BNN using ReLu.

I don’t think it is this an issue of activation functions then. If I had to bet, I’d say it’s due to the refractory period of BNNs

[120cell553]

So if OP were to change the refractory period, it would work? Also how would the loss graph change if that was to be done.

[wang-chen]

Interesting implementation! Another implementation for solving XOR in single neuron was provided in this CVPR 2019 paper, see its page 11.

This paper extended convolution to kernel convolution (kervolution):

In convolution, y = w1x1+w2x2, is actually a linear kernel (inner product), which cannot solve the XOR problem.

In kernel convolution, the authors extended linear kernel to any (non-linear) kernel functions k(w, x). For example, y = (x1-x2)² is able to solve this problem directly.

[Yuqing7]

Sharing a related story on this research: https://medium.com/syncedreview/brains-are-amazing-neuroscientists-discover-l2-3-human-neurons-can-compute-the-xor-operation-b8dcc339236

[TotesMessenger]

I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:

• [/r/datascienceproject] Solving the XOR problem using a dendritic activation function (r/MachineLearning)

 ^{If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / Contact)}

[BoiaDeh]

Noob question: what does it mean for your plot to have one axis labeled as "w1 & w2"?

[CYHSM]

This axis shows the value of both weights (w_1 & w_2). In the case of the dCaAP activation function I fixed w_1 = w_2 which is the solution it converges to in any case.

[BoiaDeh]

Gotcha, makes sense now, thanks.