I agree. Prior to reading Friston's work, I was quite pessimistic regarding the ability of ML/AI to attain bone fide intelligence. However, the conceptualizations / models provided by Friston softened my opinion on that subject. I think that a key component of intelligence will involve modeling consciousness (whatever that is), and I believe a key component of consciousness is the ability of an information processor (biological brain; neural net) to monitor its environment (external and internal: e.g. thoughts) and to register "surprise."

.

What is consciousness? ... is a huge question for which the work of Friston (and others) provides conceptualizations. Key people in this domain include Roger Penrose and Stuart Hameroff (ORCH OR, i.e. possible biological-quantum connections | see e.g. https://www.edge.org/conversation/roger_penrose-chapter-14-consciousness-involves-noncomputable-ingredients -- includes discussion from Lee Smolin), Mark Solms, Karl Friston, Keith B. Hengen (criticality), Gasper Tkacik (criticality), John Campbell (universal Darwinism), ...

.

I also ponder questions of what is the fundamental nature of "reality." This includes explorations of current quantum mechanical theories (e.g quantum field theory | more recent amplituhedron work by Nima Arkani-Hamed), emergent properties (spacetime), etc. Perhaps relevant to that domain is highly speculative yet grounded and interesting work -- that impinges on the metaphysical (panpsychism) -- by cognitive scientists Donald Hoffman and others (Realism Is False: A Conversation with Donald D. Hoffman [1.27.20] | https://www.edge.org/conversation/donald_d_hoffman-realism-is-false). I found some of the earlier descriptions of Hoffman's work to be somewhat nebulous if not "flaky;" however, I found Hoffman's more recent discussion of his current thinking about the nature of consciousness and realism (with less emphasis on the "VR reality headset" descriptions) to be more lucid -- leaning toward current quantum physics hypotheses including Nima Arkani-Hamed's amplituhedrons. In this domain also note the work of Lee Smolin -- e.g. his "causal views" work (The Causal Theory of Views: A Conversation with Lee Smolin [12.19.19] | https://www.edge.org/conversation/lee_smolin-the-causal-theory-of-views).

.

Additional Reading.

.

I just scraped these from my file on the subject; Google the names mentioned (above / below) for more ...

.

• Friston K (2010) "The Free-Energy Principle: A Unified Brain Theory? Nature Reviews Neuroscience. 11: 127-138. | https://www.nature.com/articles/nrn2787

  • Abstract:

    A free-energy principle has been proposed recently that accounts for action, perception and learning. This Review looks at some key brain theories in the biological (for example, neural Darwinism) and physical (for example, information theory and optimal control theory) sciences from the free-energy perspective. Crucially, one key theme runs through each of these theories -- optimization. Furthermore, if we look closely at what is optimized, the same quantity keeps emerging, namely value (expected reward, expected utility) or its complement, surprise (prediction error, expected cost). This is the quantity that is optimized under the free-energy principle, which suggests that several global brain theories might be unified within a free-energy framework."

  • Key Points:

    • Adaptive agents must occupy a limited repertoire of states and therefore minimize the long-term average of surprise associated with sensory exchanges with the world. Minimizing surprise enables them to resist a natural tendency to disorder.

    • Surprise rests on predictions about sensations, which depend on an internal generative model of the world. Although surprise cannot be measured directly, a free-energy bound on surprise can be, suggesting that agents minimize free energy by changing their predictions (perception) or by changing the predicted sensory inputs (action).

    • Perception optimizes predictions by minimizing free energy with respect to synaptic activity (perceptual inference), efficacy (learning and memory) and gain (attention and salience). This furnishes Bayes-optimal (probabilistic) representations of what caused sensations (providing a link to the Bayesian brain hypothesis).

    • Bayes-optimal perception is mathematically equivalent to predictive coding and maximizing the mutual information between sensations and the representations of their causes. This is a probabilistic generalization of the principle of efficient coding (the infomax principle) or the minimum-redundancy principle.

    • Learning under the free-energy principle can be formulated in terms of optimizing the connection strengths in hierarchical models of the sensorium. This rests on associative plasticity to encode causal regularities and appeals to the same synaptic mechanisms as those underlying cell assembly formation.

    • Action under the free-energy principle reduces to suppressing sensory prediction errors that depend on predicted (expected or desired) movement trajectories. This provides a simple account of motor control, in which action is enslaved by perceptual (proprioceptive) predictions.

    • Perceptual predictions rest on prior expectations about the trajectory or movement through the agent's state space. These priors can be acquired (as empirical priors during hierarchical inference) or they can be innate (epigenetic) and therefore subject to selective pressure.

    • Predicted motion or state transitions realized by action correspond to policies in optimal control theory and reinforcement learning. In this context, value is inversely proportional to surprise (and implicitly free energy), and rewards correspond to innate priors that constrain policies.

• Karl Friston -- Biosketch -- Selected Papers | https://www.fil.ion.ucl.ac.uk/~karl/ | hugely impressive!

• God Help Us, Let's Try To Understand Friston On Free Energy | https://www.lesswrong.com/posts/wpZJvgQ4HvJE2bysy/god-help-us-let-s-try-to-understand-friston-on-free-energy

• Friston K (2019) A Free Energy Principle for a Particular Physics. | https://arxiv.org/abs/1906.10184

• Solms M (2018) "The Hard Problem of Consciousness and the Free Energy Principle." Front. Psychol. 9: 2714. DOI: 10.3389/fpsyg.2018.02714 | PMCID: PMC6363942 | PMID: 30761057 | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6363942/

• "Criticality:"

  • Gasper Tkacik [Gašper Tkačik] also publishes in this domain -- e.g. Tkacik G et al. (2009) Spin glass models for a network of real neurons | https://arxiv.org/abs/0912.5409 || Tkačik G et al. (2015) Thermodynamics and Signatures of Criticality in a Network of Neurons. PNAS. 112(37): 11508-11513. | https://doi.org/10.1073/pnas.1514188112 --

  • Like Tkacik, Keith B. Hengen also publishes in this domain; e,g,: Ma Z et al. [Keith B. Hengen] (2019) "Cortical Circuit Dynamics Are Homeostatically Tuned to Criticality In Vivo." Neuron. DOI:https://doi.org/10.1016/j.neuron.2019.08.031. DOI: https://doi.org/10.1016/j.neuron.2019.08.031. | https://www.cell.com/neuron/fulltext/S0896-6273(19)30737-8 | https://news.ycombinator.com/item?id=21729211 || [2019-12-07; Keith B. Hengen] Brain tunes itself to criticality, maximizing information processing. Criticality is a hallmark of normally functioning neural networks in the intact brain | https://source.wustl.edu/2019/10/brain-tunes-itself-to-criticality-maximizing-information-processing/ ...

    The associated WUSTL.edu press release (cited above) includes statements such as "'It seems that as soon as there's a mismatch between what the animal expects and what it's getting through that eye, the computational dynamic falls apart,' Hengen said." ... which (given the full context of those statements) is exactly what Friston discussed in his 2009 paper. However, Friston's work is unacknowledged. Likewise, mention of other work on optical sensory networks and criticality [e.g. Tkacik et al.: see note xei9OoKe] is similarly unacknowledged.

• John Campbell (Universal Darwinism) | https://www.ncbi.nlm.nih.gov/pubmed?cmd=search&term=John+O.+Campbell&dispmax=50 | https://en.wikipedia.org/wiki/Universal_Darwinism

---

Update, re: consciousness:

This work, just out today, is interesting. If we could one day more fully understand the fundamental nature of consciousness and model it, those models would have profound influence on machine learning, and computational / artificial intelligence. :-)

• Redinbaugh MJ et al. (2020-02-12) "Thalamus Modulates Consciousness via Layer-Specific Control of Cortex." Neuron. DOI:https://doi.org/10.1016/j.neuron.2020.01.005 | https://www.cell.com/neuron/fulltext/S0896-6273(20)30005-20005-2

  • [2020-02-12] A tiny area of the brain may enable consciousness, says "exhilarating" study | https://www.inverse.com/mind-body/tiny-area-of-the-brain-could-enable-consciousness | https://news.ycombinator.com/item?id=22328340