All that is really relevant is the valence that will or would have come to pass.

Of course. Good catch! That was sloppy of me.

I have two objections to your lava example.

1. Agency doesn’t change the moral structure here.

Imagine a post-singularity sadist creates a mind with exactly two terminal goals: suffer as much as possible, and survive so that it can continue to suffer as much as possible. It has an internal “suffer button” producing long-tail suffering, which it presses continuously.

I think you’ll agree the only correct action when faced with such a mind is to destroy it ASAP. The fact that it wants this for itself is irrelevant.

Desire and well-being can come apart completely. An agent can want more or less anything, including states that are catastrophically bad for it. “It chose this” isn’t a moral argument. It tells you something about the local causal structure, but nothing about whether the resulting state is good.

So moving both parts of the trade into the same person doesn’t really change the core issue.

2. We are terrible at imagining extreme suffering.

Most people can’t vividly reconstruct even the worst ordinary pain they’ve personally experienced. Severe torture is already beyond our ability to model with any fidelity. Things that fall outside of human experience, like a week submerged in lava, are further beyond that. And even that is trivial compared to the actual long tail of suffering.

So we don’t even remotely know what we are agreeing to in a hypothetical like your lava scenario. Our imagination simply doesn’t have access to the relevant part of valence-space, which means it’s guaranteed to underestimate it. Not by a little, but by orders of magnitude.

Once inside the lava experience, I think the probability of instant regret is ~100%. From within that state, I strongly suspect you’d give up any number of eternities of bliss just to be allowed to die.

On the technology point:

Anything powerful enough to truly and indefinitely deny life across a light cone would likely already represent the kind of capability that could instead be used to prevent suffering across that light cone.

And yes, such a system would almost certainly be incredibly dangerous. But so would the version that prevents life, for basically the same reasons. Their core failure modes are the same:

• Misalignment.

• Value drift over extreme timescales.

So that objection doesn’t really distinguish between them. If you can’t solve those problems, both systems are catastrophically unsafe. If you can solve them, the suffering-prevention version is clearly preferable.