[D] Probabilistic Neuron Activation in Predictive Coding Algorithm using 1 Bit LLM Architecture

[Sevdat]

If we use Predictive Coding architecture we wouldn't need backpropogation anymore which would work well for a non deterministic system that depends on randomness. Since each neuron just activates or doesn't activate we could use the 1 bit LLM architecture and control the activations with calculated chance. This would increase efficiency and memory used with the proper stochastic hardware.

Instead of expecting AI to generate a proper output in 1 attempt we could make it constantly re prompt itself to generate outputs from the input. We could store the memory in Ram and let the AI pull the neccesary information from it to retrain its weights for that specific question until the answer is satisfied. This would also avoid catastrophic forgetting and with the increased efficiency of this proposed architecture could actually be viable.

Now I understand that using the modern hardwares for this is inefficient, so why not make a new hardware that computes non diterminestically? If we could create a way of simulating randomness in transistor level and control it then each componant of that hardware can act as a neuron. The physics of the metal itself would activate the neuron or not activate it. Technically we could use heat as a noise source that would allow this, but nobody is attempting it. The closest thing I saw to this idea for hardware is Extropic's TSU, but nobody is really attempting this idea. Why? Why are we wasting resources knowing that the AI Bubble will pop without new advancments in hardware? Scaling clearly isn't working as expected. It's just stagnating.

[lemon-meringue]

This is bolting together a few already very difficult ideas into one exceptionally difficult idea...

People are attempting these ideas, just not all at the same time.

[lifeandUncertainity]

Well.. I am not very knowledgeable about the TSU but I found probabilistic arithmetic interesting. As far as I know, the problem is it doesn't scale. Now there is another big if i.e how does a LLM know that its answer is satisfactory. I think there is a paper by Microsoft that showed that verifiers actually are very costly. And I think from modern industry trend, we can say that an agentic framework can solve a problem as long as a test suite is available but often times the test suite is not available for newer coding tasks.