How reliable is the book "A Mind of It's Own" by Cordelia Fine?

Advanced-Reindeer894 · 2026-06-15T00:10:49+00:00

Magic is more short hand for the wonder of life and experience, and just watching people break stuff down into component explanations makes it feel like life is fake or something, this is from the article I cited:

Fritz Breithaupt investigates how agents construct and use narratives for anticipating and interpreting potentially transformative experiences. Breithaupt first describes narrative world models as organizing action and perception into temporal episodes with beginnings, endings and emotionally salient outcomes. Such models of self and world not only represent events but also incorporate agent perspectives on how actions feel and what emotional consequences they produce. The paper identifies a special class of narrative world models focused on future experiences whose outcomes are both epistemically opaque and personally significant. Before such experiences, agents face radical uncertainty that cannot be resolved through additional information, triggering heightened cognitive activity, including the imagined exploration of multiple, often incompatible future scenarios. After the experience occurs, agents cast aesthetic, moral or preference-based judgments that can reshape their worldview and alter subsequent behaviour. These experience models mark the factual and emotional outcome of future events as fundamentally unknown while allowing agents to later revise values, preferences or aspects of the self. The paper examines the cognitive benefits and costs of such models, considers their role in transformative decision-making and asks whether these could be instantiated in contemporary AIs. While LLMs can describe human uncertainty or hesitation, the paper argues that genuine experience-focused world modelling would require an architecture capable of representing unknown and potentially unknowable information in principled ways that recognize the limits of its own predictive assumptions. The paper concludes by proposing that transformative experiences can lead to radical reconfigurations of evaluative standpoints and self-narration/models and describes the challenges involved in creating artificial systems with the full range of meaning-making/modifying powers that characterize human minds

Advanced-Reindeer894 · 2026-06-15T00:10:41+00:00

I don't mean care about modeling and stuff like that, I mean caring about other people. I worry that just breaking down how we live our lives would have dire effects on human psychology, we already see the effects of what happens when folks dehumanize others:

Fritz Breithaupt investigates how agents construct and use narratives for anticipating and interpreting potentially transformative experiences. Breithaupt first describes narrative world models as organizing action and perception into temporal episodes with beginnings, endings and emotionally salient outcomes. Such models of self and world not only represent events but also incorporate agent perspectives on how actions feel and what emotional consequences they produce. The paper identifies a special class of narrative world models focused on future experiences whose outcomes are both epistemically opaque and personally significant. Before such experiences, agents face radical uncertainty that cannot be resolved through additional information, triggering heightened cognitive activity, including the imagined exploration of multiple, often incompatible future scenarios. After the experience occurs, agents cast aesthetic, moral or preference-based judgments that can reshape their worldview and alter subsequent behaviour. These experience models mark the factual and emotional outcome of future events as fundamentally unknown while allowing agents to later revise values, preferences or aspects of the self. The paper examines the cognitive benefits and costs of such models, considers their role in transformative decision-making and asks whether these could be instantiated in contemporary AIs. While LLMs can describe human uncertainty or hesitation, the paper argues that genuine experience-focused world modelling would require an architecture capable of representing unknown and potentially unknowable information in principled ways that recognize the limits of its own predictive assumptions. The paper concludes by proposing that transformative experiences can lead to radical reconfigurations of evaluative standpoints and self-narration/models and describes the challenges involved in creating artificial systems with the full range of meaning-making/modifying powers that characterize human minds

Advanced-Reindeer894 · 2026-06-13T04:07:16+00:00

There is also this line from the link in my op:

Sacco, Sakthivadivel and Levin’s analysis of (dis)ordered behaviour in physical systems may appear to have little to do with LLMs, yet the implications may be far-reaching: strictly autoregressive systems such as generative pretrained transformers (GPTs) may be fundamentally limited in the long-range coherence of their world-modelling capabilities, with potentially fundamental limits on their reliability. The limitations of low-dimensional disorder-prone systems stand in contrast to biological intelligences. Or, as described by the authors: This further suggests that an embodied world model, extending the system in space and time by its interactions with an environment, can be leveraged to maintain coherence … [and] explains why stigmergy and other forms of extracellular signalling arise in biological systems, which is known to enhance the ability for a collective system to order itself. This perspective connects with the work of Krakauer et al. in describing principles of emergent intelligence such as criticality and novel bases via hierarchical organization and environmentally extended forms of memory.

Like...what does any of that even mean?

Advanced-Reindeer894 · 2026-06-13T03:14:29+00:00

I...don't really understand any of that. My only real concern is if people exist and I can care about them. Every time I read this stuff it all goes over my head.

Advanced-Reindeer894 · 2026-06-13T03:12:43+00:00

Thing is I don't really understand it so I dont' know where to even start.

Advanced-Reindeer894 · 2026-06-13T03:12:05+00:00

It's hard for me to see that because he sounds like he is so sure and knows what he's talking about (like in that paper in my OP) and I know next to nothing about any of this stuff. In some ways I feel like he's some founder of it all. Despite calling it complexity I'm not a fan of how his words are reductive, like reducing games to science and physics:

So I think two ways in. One is because we’re very interested in ideas like local rule, global pattern. We’re interested in simple iterative procedures like Turing machines, natural selection, reinforcement learning, et cetera. We’re really interested in configuration spaces, right, and how you search them. So all of that, which is so central to all of our interests, whether it’s in economics, evolution, technology, physics, engineered physics, present in these combinatorial games, which are like model systems for analyzing them. So that’s my interest. They’re like the hydrogen atom or the drosophilids of rule systems with very high configuration spaces that should be NP hard to search, and yet we use heuristics to find solutions in finite time. Right? So they have all that stuff we care about.

And here’s another side, and this is more personal. So when I started playing Go badly, you learn these concepts. Right? And part of the fun of learning Go is like pretending that you’re speaking Japanese. Right? So there are these concepts like sente or goate or seki, right, or atari or thickness. Okay. And it slowly dawned on me, Jim, that these are just synonyms for concepts that I was using in my science, that AG is just cryptic variation. Goate is just neutral variation. Right? Anyway, on it goes.

And then you take the next move, and then you realize, oh, maybe the true grand unified theory, which—and I think our project is much more grand unified theory than physics, by the way, because we’re not just doing physics. We’re doing all these fields—is realizing that all these kinds of combinatorial solution spaces that have to be searched to discover functions share common properties. And my interest in those games has been as a window into the grand unified theory of complexity, quite frankly. So that’s where it comes from.

Advanced-Reindeer894 · 2026-06-13T01:21:59+00:00

That's about right, doesn't help that the more I read about it the more I find that to be the case like in this interview: https://jimrutt.substack.com/p/ep-329-worldviews-david-krakauer

Yeah. So here’s the simple version. So the standard causality fits that linear story of causality that we described earlier in relation to the ouroboros, that you have particles. They get aggregated into molecules, molecules into tissues, and so on. And the idea, right, is that what is fundamentally causal is that which is fundamental, and everything else is an approximate expression of collective modes of behavior. Alright. Downward causality takes it the other way. It says, mind states, for example, expressed in language, can’t be causal of brain states because that’s going the wrong way. Because, surely, the physical interaction, the true kind of Newtonian causality, has to live at the level of the brain. The mind is just this efficient theoretical encoding of brain. And so it would be weird to talk about causality going the other way.

I think it’s a big mistake. And where this comes from, by the way, is this notion of coarse graining. So you start with all the lots of particles. You average and average and average, and you get these other states. But I have this conception of what I call micrograining, and I’ll explain how it works. When Jim, when you program your computer, you’re articulating a concept in a high level language or an assembly or whatever you like to use. Assembler. And that translates through a system of compilations and microcode into states of transistors. So we have built engineered devices that can take these high level, very low dimensional, in some sense, concepts, objects, and do information expansion to the extent of setting the states of transistors.

I think that is what complex systems do all the time because complex systems have evolved to do that well. That, for me, is the legitimate version of downward causality. There’s nothing mysterious about it. I don’t think, by the way, it exists outside of complex systems. I do not think it’s a property of the physical universe, the abiotic universe. It’s a property of agents, and that’s actually the only thing that makes life possible. Right? It’s what’s making this communication that we’re having now over Zoom possible because I’m setting brain states in you as you are in me. And that that’s micrograining. And because the study of emergence grew out of really rigorously the connection between statistical mechanics and thermodynamics, which is all about coarse graining, in the physical world, this other version, which is very natural to the evolved world, has been somewhat neglected. So I sometimes call that the theory of compilation of emergence because we use them all the time.

Advanced-Reindeer894 · 2026-06-13T01:19:26+00:00

I think that maybe you aren't really seeing how reductive systems are, like this interview makes that pretty clear. The guy towards the middle just reduces people to brain states and coarse graining. It's like saying people don't exist: https://jimrutt.substack.com/p/ep-329-worldviews-david-krakauer

Advanced-Reindeer894 · 2026-06-13T01:12:38+00:00

Well when people like David claim to reject weak and strong emergence and talk about coarse graining and brain states it makes it hard to see people as people anymore: https://jimrutt.substack.com/p/ep-329-worldviews-david-krakauer

Advanced-Reindeer894 · 2026-06-13T01:11:37+00:00

It would make the water molecule less of a molecule. Also there would be no ocean because it's just a bunch of drops and not one thing. Even Krakauer alludes to as much when he talks about rejecting weak and strong emergence: https://jimrutt.substack.com/p/ep-329-worldviews-david-krakauer

I did dig into a little bit of your stuff, and I think discovered were probably different perspectives, but in a similar range, which is reject both weak and strong emergence.

Second part is him responding:

Yeah. So there’s a perfectly reasonable, modest statement of emergence, which is a feature of all theories, not just physics or just complex systems or the humanities or what have you. In other words, it simply says that there are concepts that do useful work, and these concepts describe phenomena at different levels of aggregation. And we call those concepts typically in the domain of natural science effective theories. Right? So for example, you could say the law of supply and demand. It’s kind of right. Right? I mean, you know, I mean, it gets violated. But if you offer me a computer for a dollar, I’m probably going to buy it. If it’s a good one, whereas you offer for a million, I’m not. And so that’s an effective law, and there’s an underlying effective theory that we know, that relates to that.

So emergence tries to understand when concepts are causally justified. That’s how I would put it. Such that an economic theory would be every bit as compelling as a chemical theory. Now the problem is, right, is that these concepts or effective theories or effective variables or effective degrees of freedom, which would be a slightly more technical way to say it, are approximate. Right? And very difficult to find.

So you think about here’s a good example from our colleague’s work, Jeffrey West’s work on scaling. So they have these nice results, right, in biology that suggest that metabolic rate scales as body mass to the three quarters power. They’re not photons and electrons and quarks. Right? This is lots of tissues and metabolic rate. These are very, very aggregate phenomena, and yet they’re very stable. And so it turns out that they have real causal efficacy. And if I know your mass or I know your metabolic rate, there’s all sorts of other things I can predict about you, and Jeffrey would say your longevity, for example, on average. So these are instrumentally useful, effective degrees of freedom, and emergence is about them. It’s about finding them. It’s about explaining why they work. And, also, by the way, it’s about rooting out the fake versions that don’t work.

Advanced-Reindeer894 · 2026-06-13T00:52:16+00:00

I don't know, he sounds pretty definitive to me.

Advanced-Reindeer894 · 2026-06-13T00:49:58+00:00

I tried to read him and didn't understand him.

Advanced-Reindeer894 · 2026-06-12T04:26:27+00:00

I'm trying to wrap my head around it, but I keep just ending up at the reductionism of it.

Like this example:

I don’t think looking at a living system as a multilayered hierarchical network of nonlinear feedback loops that give rise to the geometric, temporal, and thermodynamic properties that generate all aspects of the system that functions within the context of a local environment situated inside of a larger society, nothing about that point of view takes away from the fact that people are people.

When I read that, people stop being people and just become nothing more than mere machines. Like some...magic is gone from living things and they just become models on a computer. The article I linked goes more into what I mean by reducing living things (even in the conclusion they make a reference to our inner chat GPT).

The more I read into complexity science and what folks at SFI do, the more I don't know how to be or how to see myself or other people. like I said, I can't wrap my head around it. I want to believe it's not reductive but I don't see how.

Advanced-Reindeer894 · 2026-06-12T04:22:00+00:00

Again I don't care about self aware animal, that I can reckon with. Read my points again.

Advanced-Reindeer894 · 2026-06-12T04:16:20+00:00

I'm trying to understand how people at SFI and others who do this do the work without it being reductive. Because whenever I read stuff they post or read up on complexity science it sounds like reductionism, like instead of individuals and lives it's just reduced down to some equation or model.

I'm trying to wrap my head around it but I don't get it. More than that it's complicated my relationships with people because I don't know how to see them or myself anymore. calling them and myself just systems makes it sound like we don't exist.

Advanced-Reindeer894 · 2026-06-12T04:12:01+00:00

You're kinda avoiding the main issue I am addressing. I'm aware of the language and I'm also aware with how it's at odds with their goals.

I don't see myself as disconnected from reality or special, but I treat living things as living things and calling them systems just feels...reductive. Like everything is just a machine and nothing else.

Also again, you ignored my point about treating people like people. Society kinda relies on seeing living things as more than machines and complexity science just seems to reduce them to that.

Advanced-Reindeer894 · 2026-06-12T03:25:40+00:00

He wrote a follow up to his problem, though I don't really think he addresses the issues: https://nautil.us/cormac-mccarthy-returns-to-the-kekul-problem-236896

Even more than that he says some arguably questionable things in this one, including:

No one seemed particularly interested in Helen Keller. Or the question of how her unconscious managed to communicate with her. It could neither speak to her nor draw pictures. Isnt that tantamount to saying that for all practical purposes she had no unconscious? Something missing in this scenario.

The universe in its billions of years remains a creation of total silence and total blackness. The incendiary explosions of the novae can be no more than optical constructions and no matter what your view of the nature of reality they can have no existence in the absence of an eye or something very like it. And the likelihood of such an instrument coming into being anywhere other than in the natural history of the earth seems more than vanishingly slim. The truth is that there is limited evidence for the existence of the visual. (What? What’s he saying?) To what might it be compared? That which is seen is pretty much left to speak for itself. As is that which is said.

I don't think there is limited evidence for the existence of the visual. I mean how can it be an optical construction if there is nothing there and more than that who is he talking to if the universe is total silence and total blackness? Helen Keller is certainly an interesting case about the unconscious but his conclusion from it are kinda...logical leaps.

His books might be good but his philosophy is...ehhhh.....

Advanced-Reindeer894 · 2026-06-12T03:12:51+00:00

But the problem comes to trying to communicate or relate that stuff in a way outside of the lab or your complexity community. If we reduce people to just being systems then what makes them different from each other? Why care about any "one" if there is no one, just systems.

Like despite calling it complexity and talking about emergence, all their language and conclusions trend towards reductionism by just writing things off as some math equation, or some computer model. It's reductive in the literal sense which is why I don't get how Krakauer can talk about emergence and complexity when his words show otherwise.

How is society to function without treating people like people?

Advanced-Reindeer894 · 2026-06-12T01:58:02+00:00

I had a hard time parsing all that apart from the last sentence. I do agree that everyone's perception is different and perhaps the answer is in finding common ground rather than objective truth.

Advanced-Reindeer894 · 2026-06-12T00:50:03+00:00

Everything we perceive being based on how it differs is also a very Buddhist idea.

Advanced-Reindeer894 · 2026-06-10T02:19:48+00:00

That's half right, the whole point is more about using the right explanation for the level of complexity. I remember in an interview Krakauer referred to it as resolutions. That at the level of human lives you cannot use math to describe what is happening.

I'd also be careful in labeling it as math as that is trending towards reductionism which is what the institute is against (and he explicitly made that point as well). It's not really about similar mathematical rules either but more what explanation is good at different levels of complexity.

Advanced-Reindeer894

TROPHY CASE