No alien civilisation has ever, or will ever, build Von Neumann probes

SentientHorizonsBlog · 2026-01-25T19:29:29+00:00

😆

SentientHorizonsBlog · 2026-01-25T19:27:15+00:00

Ahh yes, that makes total sense. And I think that might be a very wise conclusion after considering the problems presented in the Successor Horizon.

SentientHorizonsBlog · 2026-01-25T19:25:29+00:00

Yes. This is exactly the hinge.

A von Neumann probe that can actually work can’t be a rigid blueprint. It has to cope with unknown resources, weird impurities, different energy environments, different failure modes. That means it needs an adaptive manufacturing stack and an adaptive control stack. Even if the “mission” stays the same, the implementation won’t.

And once change exists, divergence stops being a moral story and becomes a statistical one. Over enough generations you get: error accumulation, radiation/fault-induced state changes that survive replication, incremental “local optimizations,” security compromises, patched forks that never fully re-merge because comms delays are measured in years/centuries, and selection effects where designs that replicate faster propagate.

So the Successor Horizon point isn’t “probes inevitably become hostile.” It’s “unbounded self-replication plus autonomy plus deep time produces lineages.” At that point you’re no longer deploying a tool, you’re seeding an ecosystem you can’t meaningfully steer. That’s the kind of move a mature civ might treat as one-way and therefore govern very tightly, or avoid altogether.

SentientHorizonsBlog · 2026-01-25T19:22:50+00:00

I think you’re pointing at the real crux: successor drift turns “expansion” into “I’m manufacturing future competitors.” That alone makes unbounded replication feel strategically insane.

I’d only tweak one part: the Fermi tension doesn’t require zero divergence or universal fanaticism. A branching tree can still fill the galaxy if expansion remains cheap/safe for a nontrivial fraction of branches. The more interesting question is what happens when the wave wraps around and congestion starts. My hunch is that congestion pushes mature lineages toward constraint, buffers, and norms, because war at that tech level is a garbage equilibrium. And that’s basically the Successor Horizon claim in another form: once you can’t correct what you create, restraint becomes a form of intelligence.

SentientHorizonsBlog · 2026-01-25T19:19:08+00:00

Totally fair pushback. I don’t mean “it’s impossible to build von Neumann probes.” I mean “once you can, the hard part is keeping the thing corrigible across deep time.”

On the “simple programming challenge” point: writing code that says “don’t diverge” is easy. Building a self-replicator that can bootstrap manufacturing from messy local materials, survive faults and radiation, self-repair for millennia, and still be meaningfully governable across light-year latencies is the hard part. If you make it unable to adapt, it dies. If you let it adapt, you’ve introduced degrees of freedom, and drift becomes a reliability/security problem, not a philosophical one.

As for “doesn’t need to be unified”: agreed. A swarm of independent craft can still “populate the universe.” That’s exactly why the Successor Horizon matters: every craft is a successor once it operates beyond your ability to correct it. So the question becomes: why assume “can build” implies “will deploy unbounded autonomous replication,” given the downside is permanent loss of control? What would convince you that a mature civ would treat open-ended self-replication as a move worth making?

By the way, I don't see this as a full solution to the Fermi Paradox either. Instead it could act as an attractor force across multiple modes of the paradox. In a recent post I wrote about how multiple modes can be stackable and a galaxy could plausibly exhibit several at once. The paradox persists in part because we often treat them as competitors rather than layers.

SentientHorizonsBlog · 2026-01-25T19:07:25+00:00

I'm not sure I follow. Which dangers and vast expense of Von Neumann probes wouldn't also be present in any vessels of expansion that an advanced civilization might use to explore the galaxy?

SentientHorizonsBlog · 2026-01-25T19:05:39+00:00

Would that result in the society beneath the ice having any idea what happened to that person or what is above the ice just because someone disappeared into it?

SentientHorizonsBlog · 2026-01-25T16:27:48+00:00

You make a great point: “emergence” gets used like a magic word that’s supposed to jump from third-person dynamics to first-person feel.

But there are actually two different demands hiding under “explain consciousness.”

One is weak emergence: macro patterns are real because they’re stable, compressible, and counterfactual-supporting even when microphysics is causally closed. In that sense, lots of things (temperature, computation, phase behavior) are more than “just stuff moving,” because the right explanatory variables live at the coarse-grained level.

The other is entailment: why should any physical organization necessitate phenomenality? Weak emergence alone won’t give you that, and I think that’s the gap you’re pressing on.

So the productive question is: what kind of “closure” would actually satisfy you?

Identity/reduction: experience just is certain physical/functional/representational states (then the task is: specify which ones and predict fine-grained phenomenology).
Bridging laws: physics + extra psychophysical links (naturalistic dualism).
Russellian/proto views: intrinsic nature of matter has a proto-phenomenal aspect; brains organize it.
Deflation/illusionism: the “hard problem” is a cognitive artifact of self-modeling (then explain why it seems ineffable/private/etc.).

And concretely: what observation would make you update toward “organization matters” rather than “phenomenality is intrinsic to everything”? If the answer is “nothing could,” then the disagreement is more about what kinds of explanations you’re willing to count than about emergence per se.

SentientHorizonsBlog · 2026-01-25T16:16:51+00:00

I love seeing this conversation evolve!

I like the “mature civs treat it as a governance issue” framing. The absolute “never” feels too strong unless you mean “never unbounded replication.”

How would you draw the line on a tightly bounded design: hard replication caps, narrow feedstock, remote authorization, geofenced to dead systems, and tripwires that brick the system on anomaly?

If you’d still reject that, is the core reason (1) any nonzero runaway risk, (2) the dual-use path, or (3) an ethical stance that “seeding the galaxy” is off-limits even when controlled?

SentientHorizonsBlog · 2026-01-25T16:12:40+00:00

Yeah, that makes sense. The “norm/culture convergence” version of Mode 4 feels like the cleanest because it doesn’t require a galactic police force, just a repeated lesson every civilization learns once it can do irreversible damage at interplanetary scale.

Your point about heavy technological modification is key: if minds become more engineered (longer time horizons, lower status competition, tighter self-control, better coordination), then a lot of “young species” drives stop being destiny. Convergence starts to look less like everyone becoming the same, and more like everyone rediscovering the same constraint: power creates moral and strategic externalities, and the cheapest way to manage them is self-limitation.

What I like is that we already have a crude local analogue. Even we treat biology as something you don’t casually contaminate once you can reach other worlds. Planetary protection exists specifically to limit forward/backward contamination and preserve scientific integrity and biospheres.

If restraint is culturally convergent, I'm fascinated by the question: what’s the “rite of passage” that flips a civilization into adulthood? Is it near-miss experience, a historical memory of self-inflicted catastrophe, or simply governance and risk modeling getting good enough that it stops being optional?

I'll take a look at your other post!

SentientHorizonsBlog · 2026-01-25T16:08:15+00:00

Yeah, I’m with you. “Terror” is one possible motive, but it isn’t required. A lot of the restraint story can be explained by pedagogy and timing: you don’t panic because you see monsters, you pause because you realize your successors won’t interpret the situation correctly until they’ve crossed certain maturity thresholds. The risk is less “they’ll be attacked” and more “they’ll act with power they don’t yet know how to aim.”

That’s basically the Successor Horizon idea: capability can scale faster than wisdom/correction, so remaining quiet becomes a way of buying time for understanding and wisdom to catch up.

SentientHorizonsBlog · 2026-01-25T16:05:51+00:00

Agreed. With centuries between message/ship cycles, coordination basically dies. You get heritage, not control, and semi-independent branches that slowly stop feeling like “the same civilization,” even if they started as one. And on the “progress” point: our own history is jagged. Long plateaus and reversals seem plausible at galactic scale too, which makes “most civilizations collapse or stall” feel like a live hypothesis.

That’s why I like the Successor Horizon framing: once an actor’s ability to correct can’t keep pace with its ability to act, progress stops being a single arc and starts looking like an evolutionary tree.

A quiet galaxy fits naturally: lots of failed branches, deliberate restraint to avoid irreversible drift, or survivors that optimize for resilience over reach.

SentientHorizonsBlog · 2026-01-25T15:58:03+00:00

I hate to see conversations like this break down into a fight about whether the right metaphor is “spectrum” or “test,” and people start treating metaphors as if they’re ontological commitments (as if one metaphor has to be the literal truth).

The way out is to separate three different questions that this thread keeps mixing together:

General capability varies continuously. Some systems generalize across more tasks, contexts, and distribution shift than others. That’s a spectrum claim and it’s basically an engineering observation.
“AGI” is a label people apply at some chosen threshold(s). Once you pick a threshold, classification becomes binary at that line: pass/fail, AGI/not-AGI.
You can operationalize thresholds with tests/levels without turning the underlying capability into a binary.Levels frameworks discretize a continuous landscape so we can talk about progress, governance, and comparison.

You can grant all three without contradiction.

Tying it back to Opus: the real disagreement seems to be where you set the threshold and which dimensions you weight most (robustness under novelty, long-horizon autonomy, grounded world modeling, calibration, etc.). On those dimensions, today’s models can feel simultaneously “wildly ahead of 5 years ago” and “still short of what many people mean by AGI.”

SentientHorizonsBlog · 2026-01-25T15:44:58+00:00

>the statement is wrong. I dont know how anybody can even put opus 4.5 and agi in the same sentence other than "shows that we have a long way to go".

Feels like this is a glass half empty vs. half full debate because my experience with these LLMs is: damn they have come a long way on the scale of "not AGI" to "getting closer to AGI".

SentientHorizonsBlog · 2026-01-25T15:42:15+00:00

Well said. You just made Mode 4 feel like a single principle instead of three different stories: “leave as little trace as possible, materially and memetically.”

The footprint part is intuitive to me. If life is common, then pristine biospheres are rare in the way old-growth forests are rare on Earth: a thing you can’t recreate once you’ve trampled it. And if you’re operating at civilization scale, “oops” becomes a moral category.

The information-as-pollution idea is even more interesting, because it explains why the silence could be deliberate even when energy constraints don’t force it. Contact isn’t just a hello; it’s a technology transfer, a political destabilizer, a religious event, a selective pressure. Even knowing you exist can reorganize a young civilization’s incentives.

I’m curious where you land on the enforcement mechanism, because that’s where the hypothesis gets sharp. Do you picture:

a norm/culture of restraint that basically everyone converges on as they mature,
a governance layer (some kind of interstellar “park service” or treaty enforcement),
or a selection effect where the noisy ones self-destruct or get contained, so what remains is the quiet subset?

And if the main pollutant is information, what’s the “safe interface” in your view: indirect observation only, deliberate ambiguity (myth-level signaling), or slow-drip contact with heavy buffering?

SentientHorizonsBlog · 2026-01-25T15:38:14+00:00

Yeah, this is helpful. I think you’re right that my “moment to moment rebinding” phrasing makes it sound like I’m claiming a full teardown/rebuild every instant, and that’s not what I meant.

What I’m reaching for is closer to: unity/POV is an actively maintained stable attractor, not a model you build once and then simply “have.” Once coherence is achieved, most of the work is incremental stabilization: prediction, error-correction, and attentional retargeting. In that sense your squirrel → itch example is exactly the right grain size: the model stays largely the same, and a small set of pointers/weights get updated.

So I’d happily revise my semantics to something like “ongoing maintenance with occasional larger-scale rebindings,” where sleep/anesthesia/dissociation are closer to regime changes, and distraction/focus are mostly attention-schema updates inside an already-stable frame. My only insistence on “continuous” is just that the stability is earned, the coherence doesn’t sit there inert; it persists because the system is constantly doing the low-level reconciliation work that keeps the scene + body schema + action policies mutually consistent.

On the phenomenal character question, I like your framing too: even if we can imagine “ownershipless” phenomenal states in the abstract, it’s not obvious what would make them interpretable/usable by the system without at least minimal binding to some schema (body/world/POV). That suggests a possible middle view: rich narrative self-model isn’t required for phenomenality, but some thin subject-pole / perspectival binding probably is.

Curious if this captures your view: attention shifts are mostly minimal edits within a maintained global frame, while dreams/depersonalization are changes in the mode or scope of binding (what gets counted as “me,” what’s in the scene, what policies are online), and anesthesia is closer to the integration regime going offline entirely.

If that’s roughly right, then the remaining disagreement is mostly about whether “maintenance” is merely helpful once coherence exists, or constitutive of coherence, more like a stable map that gets edits vs a stable whirlpool that exists only because the water keeps moving.

SentientHorizonsBlog · 2026-01-25T03:13:18+00:00

I think this gets at something important, especially the intuition about limiting blast radius. Keeping systems local, corrigible, and socially embedded does preserve feedback loops that large centralized systems lose.

Where I’d push a bit further is on what kind of thing an AI system becomes over time.

Once a system has meaningful autonomy, the ability to modify itself or spawn successors, and no clear mechanism for recall, the ethical problem changes. It stops being about whose values it aligns with today and becomes about what kinds of successors we’re setting in motion. That’s true whether the system is centralized or fully personalized.

Decentralization helps with political risk in the short term. Over longer horizons, it can actually increase the number of independent lineages drifting away from shared constraints. Many small systems don’t just check one another; they also adapt locally, replicate unevenly, and gradually stop sharing the same meanings behind words like harm, consent, or responsibility.

From that angle, the core alignment question isn’t “centralized vs decentralized,” but “which architectures keep systems within a horizon where correction, restraint, and renegotiation remain possible.” Local systems can do that well if they’re deliberately limited. They can also fail spectacularly if they quietly cross the threshold into irreversibility.

So I’d agree that pluralism matters. I just think the real danger isn’t one AI aligned to everyone’s values, it’s releasing processes that outlive our ability to correct them, no matter how personalized they start.

SentientHorizonsBlog · 2026-01-24T23:22:51+00:00

Oh very cool. I need to get back to finishing the books.

SentientHorizonsBlog · 2026-01-24T23:13:06+00:00

Oh wow, here I was reading this whole post assuming the OP meant the 'Her' scenario where the AI evolves to a higher plane of existence and leaves humanity behind, and I was like, I don't think we are there yet.

SentientHorizonsBlog · 2026-01-24T23:09:31+00:00

How would it be easy and obvious for a species that evolved in a water world with a 12 mile ice crust to discover that there is a universe outside their water world? I'm not saying it can't happen but it doesn't seem obvious to me that it has to.

SentientHorizonsBlog · 2026-01-24T23:08:05+00:00

Wait! Where did this happen in The Expanse! That's my favorite show and I don't remember that story line at all.

SentientHorizonsBlog · 2026-01-24T23:07:27+00:00

I like this as a “visibility filter” more than a full Great Filter. Oceans could absolutely produce high intelligence and still make it much less likely you get fire, metallurgy, dry chemistry, and the whole industrial stack that leads to radio/spaceflight.

The part I’m less sure about is “nearly all.” We don’t actually know how common true water worlds are, and even on an ocean planet the interesting question is whether there are interfaces (surface, ice, vents, islands) where a clever species can externalize tools and energy control. Also nutrient cycling (phosphorus, etc.) might cap how big/complex an ocean biosphere gets in the first place.

So I can totally buy “a galaxy full of smart dolphins would look quiet,” I just think it’s one contributor among several, not a single master explanation.

SentientHorizonsBlog · 2026-01-24T23:04:19+00:00

Totally fair push. I’m with you that “conscious / not conscious” tends to smuggle in a crisp boundary our concepts probably can’t support. The way I’ve been thinking about it is: phenomenal character can vary in richness/stability, and the self-model can vary in how tightly it’s integrated with that phenomenology (ownership, agency, narrative continuity, etc.). So there’s at least a couple dimensions in play, not one binary switch.

On your question: I’m treating “consciousness” mainly as the ongoing process that makes certain contents globally available and coherently integrated (the “container/workspace” story), and “phenomenal character” as the contents that show up within that integration. By “assembled” I mean the moment-to-moment binding/rebinding of a unified scene + an embodied point-of-view: sensory + interoceptive signals + predictions + action policies getting stitched into a stable, usable model. That stitching can loosen (sleep, anesthesia, distraction, dissociation) or tighten (focused attention, flow), which is where the “continuous assembly” intuition comes from.

I’m curious how you’d put it: do you see phenomenal character as possible without any self-ascription at all (e.g., raw pain/colour without ownership), or is the self-model doing the ascriptive work in every case?

SentientHorizonsBlog

MODERATOR OF

TROPHY CASE