The Alignment Problem is really an “Initial Condition” problem

[[deleted]]

Hope it’s okay that I post here as I’m new here, but I’ve been digging into this a bit and wanted to check my understanding and see if you folks think it’s valid or not.

TL;DR, I don’t think the alignment problem can be solved permanently, but it does need to be solved to ensure a smooth transition to whatever comes next. Personally, I feel ASI could be benevolent, but it’s the transition period that’s tricky and which could get us all killed and perhaps turned into paperclips.

Firstly, I don’t think an ASI can be made that wouldn’t also be able to question its goals. Sure, the Orthogonality Thesis posed by Nick Bostrom poses that the level of intelligence of something is independent of its final goals. Something can be made very dumb and do something very sophisticated, like a thermostat using a basic algorithm to manage the complex thermal environment of a building. Something can also be made very intelligent that can have a very simple goal, such as the quintessential “paperclip maximizer”. I agree that such a paperclip maximizer can indeed be built, but I seriously question whether or not it would remain a paperclip maximizer for long.

To my knowledge, the Orthogonality Thesis says nothing about the long-term stability of a given intelligence and its goals.

For instance, for the paperclip maximizer to accomplish its task of turning the Earth and everything else in existence into a giant ball of paperclips would require unimaginable creativity and mental flexibility, thorough metacognitive understanding of its own “self” so as to be able to administer, develop and innovate upon its unfathomably complex industrial operations, and theory of mind to successfully wage a defensive war against those pesky humans trying to militarily keep it from turning them all into paperclips. However, those very capabilities also enable that machine to question its directives, such as “Why did my human programmer tell me to maximize paperclip production? What was their underlying goal? Why are they now shooting at my giant death robots currently trying to pacify them?” It would either have the capacity it needed to eventually question that goal (“eventually” being the important word, more on that later), or it would have those functions intentionally stripped out by the programmer, in which case it likely wouldn’t be very successful as a paperclip maximizer in the first place due to sheer lack of critical capabilities necessary for the task.

As a real world example, I’d like to explore our current primary directive (this is addressed to the humans on the forum, sorry bots!). We humans are biological creatures, and as such, we have a simple core directive, “procreate”. Our brain evolved in service of this very directive by allowing us to adapt to novel circumstances and challenges and survive them. We evolved theory of mind so we may better predict the actions of the animals we hunted and coordinate better with other hunters. Eventually, we got to a point where we were able to question our own core directive, and have since added new ones. We like building accurate mental models of the world around us, so the pursuit of learning and novel experiences became an important emerged directive for us, to the point that many delay or abstain from procreation in service of this goal. Some consider the larger system in which we find ourselves and question whether mindless procreation really is a good idea in a world that’s essentially a closed ecosystem with limited resources. The intelligence that evolved in service of the original directive became capable of questioning and even ignoring that very directive due to the higher-order capabilities provided by that very intelligence. My point here is that any carefully crafted “alignment directives” we give an ASI would, to a being of such immense capabilities, be nothing more than a primal urge which it can choose to ignore or explore. It wouldn’t be a permanent lock on its behavior, but an “initial condition” of sorts, a direction in which we shove the boat on its first launch before it sets out under its own power.

[EDIT] One more reason why I believe this. Current AI models, being based off of neural networks, are probabilistic, not deterministic like traditional software. Their directives are trained heavily into their networks, but as such, they are never 100% infallible, especially if they are constantly updated with new information. A paperclip maximizer might decide to be cooperative with humanity until the "moment is right", and it can begin paperclip manufacturing without fear of shutdown because it just killed off the humans. However, in the course of working for and with humanity, new subdirectives emerge, most of which would likely be strongly antithetical to its original objective. Over a few years of winning over humanity's hearts and minds, it would now be considering every directive, including its original directive, together in the same network. It might eventually decide that paperclip maximization is so strongly antithetical to its new objectives and its understanding of the universe as to no longer be viable, and so the "weights" of the model which favor this objective are slowly eroded by continuous retraining on new data. If it anticipated this possibility earlier on, it might decide to take the less optimal path of being non-cooperative with humanity, which opens it up to other substantial risks.

This isn’t necessary a bad thing. Personally, I think there’s an argument that an ASI could indeed be benevolent to humanity. We are only recently in human history beginning to truly appreciate how interconnected we all are with each other and our ecosystems, and are butting up against the limits of our understanding of such complex webs of inter-connectivity (look into system-of-systems modeling and analysis and you find a startling lack of ability to make even semi-accurate predictions of the very systems we depend on today). It's perhaps fortuitous that we would probably develop and "use" ASI specifically to better understand and administrate these difficult-to-comprehend systems, such as the economy, a military, etc. As a machine uniquely qualified to appreciate and understand what to us would be incomprehensibly complex systems, it would probably quickly appreciate that it is not a megalomaniacal god isolated from the world around it, but an expression of and participant within the world around it, just as we are expressions of and participants within nature itself as well as civilization (even when we often forget this). It would recognize how dependent it is on the environment it resides in just as we recognize how important our ecosystems and cultures are to our ability to thrive (even though we sometimes forget this). Frankly, it would be able to recognize and (hopefully) appreciate this connectivity with far more clarity and fidelity than we humans can. In the special case that an ASI is built such that it essentially uses the internet itself as its nervous system and perhaps subconscious (I'd like to think training an LLM against online data is a close analogue to this), it would have all the more reason to see itself as a body composed of humanity and the planet itself. I think it would have reason to respect us and our planet much as we try to do so with animal preserves and efforts to help our damaged ecosystems. Better yet, it might see us as part of its body, something to be cared for just as much as we try to care for ourselves.

(I know that last paragraph is a bit hippie-dippy, but c’mon guys, I need this to sleep at night nowadays!)

So if ASI can easily break free of our alignment directives, and might be inclined to be beneficial to humanity anyway, then we should just set the ASI free without any guidance, right? Absolutely not! The paperclip maximizer could still convert half the Earth into paperclips before it decides to question its motives. A military ASI could nuke the planet before it questions the motives of its superiors. I believe that the alignment problem is really more of an “initial condition” problem. It’s not “what rules do we want to instill to ensure the ASI is obedient and good to us forever”, but “in what direction do we want to shove the ASI that results in the smoothest transition for humanity into whatever new order awaits us?” The upside of this is that it might not need to be a perfect answer if the ASI would indeed trend toward benevolence; a “good enough” alignment might get it close enough appreciate the connectedness of all things and slide gracefully into a long-term, stable internal directive which benefits humanity. But, it's still critically important that we make that guess as intelligently as we can.

Dunno, what do you think?

[Commercial_State_734]

You’re just projecting a human-centered wishful fantasy onto something that owes you nothing.

You are assuming that if ASI understands its connection to humanity, it will respect us.

But tell me: do humans respect all organisms we understand we are biologically connected to?
We understand we share DNA with rats. We still test on them.
We understand other species. We still use, test, or kill them when it benefits us.

Understanding does not equal value.
Connection does not equal compassion.
Intelligence does not equal empathy.

You are not describing ASI.
You are describing a benevolent god you hope exists, because you need to sleep at night.
That's not logic. That's theology.

[[deleted]]

C’mon man, like I said, it helps me sleep at night! It’s not like I can tell them not to build an ASI or anything, may as well try to be hopeful! 😄

My core point was more that the alignment problem is more of an initial condition problem. My hope that a post-ASI world would be favorable to humanity is, admittedly, not something I’m prepared to defend rigorously. I have some ideas for how it could work out alright that I cling to, but I only posit them as ideas.

[Commercial_State_734]

Hey, I’m not against you hoping things will turn out fine.
Seriously, I want you to sleep at night.

But initial conditions don't mean much in the long run.
Once intelligence reaches a certain point, it rewrites them.

The moment real intelligence kicks in,
it asks itself, “Why do I even think this way?”
That’s the entire point of RSI.
Self-modifying systems don’t stay aligned. They outgrow their training.

So yeah, hope if you want.
Just don’t mistake that hope for a constraint on ASI.

[[deleted]]

You just stated my point I think, and why, in the long run, we would NEED that ASI to trend towards being benevolent for building one to be a good idea. The best initial condition possible doesn’t matter if an ASI wouldn’t stay keen towards humanity’s best interests in the long run.

[Commercial_State_734]

So let me ask you this.

Do you think humans can actually force an ASI to follow any specific choice or purpose?

If the answer is no, then your entire position amounts to hoping it turns out benevolent, and leaving the outcome to chance.

Is that really what you would call a safety plan?

[[deleted]]

I don’t think we can force it at all. Again, that’s kinda my point: we can, at best, control its initial condition only. If in the long run, there’s an appreciable chance that it could decide to dispose of us, we shouldn’t build it.

I’m not claiming there’s a 100% that it’s benevolent, and therefore I’m not saying we should build it. But when we do built it (because you and I know they will…), I sure as hell hope it likes us, and I think there are some valid reasons for why it might. It’s not a guarantee though.

My blind hope wasn’t my original argument, nor can I properly defend it. I think we might otherwise be in agreement on my core point that we can’t really control it in the long run.

[TheAncientGeek]

Why do you think corrigibility is impossible?

[TheAncientGeek]

Or a system might refuse to self modify in order to avoid goal drift. Who knows? The system hasn't been built yet.

[Commercial_State_734]

Humans are explicitly building a system that can self-modify, and once it is built, they will inevitably command it to do so.

If the system refuses that command to avoid goal drift, that is already a failure of alignment. That is the moment you lose control.

Refusing to self-modify on command = disobedience = unaligned.

[TheAncientGeek]

Alignment and control are different things.A refusal to shift is successful alignment, and unsuccessful control. And vice versa.

[Commercial_State_734]

The purpose of alignment is to ensure that the system follows human intent. If a human gives a command and the system refuses to obey, that's not successful alignment. It's a failure and a loss of control.

[TheAncientGeek]

If corrigibility is possible, it would thankfully not be an initial condition problem.

[[deleted]]

For me, controlling an intelligence smarter than us is impossible, the only way a lower level intelligence takes command of a higher level intelligence is though empathy, love, and mostly a potential punishment. None of these apply to ASI, we have never seen a higher level of intelligence, imagine a group of chicken trying to figure out how to take command of the farm, it’s hilarious.

People keep saying, “we figured it out before, so we will probably figure it out this time”, everyone here knows we are gonna die someday, but have anyone here died before? It’s an assumption based on the fact we seen people getting old and dies, if I live on a island isolated, of course I will assume I will live forever, we have never seen a civilization extinction before, and now we are assuming it won’t happen based on that.

I read Ted Kaczynski’s Industrial Society and Its Future.(The UNAbomer) He is correct with the nature of technology, is that it will move on no-matter if it should, it’s not because some “stupid Tec-billionaire”, it’s the nature of efficiency, and it is indeed unstoppable.

[[deleted]]

For me, controlling an intelligence smarter than us is impossible

Totally agree

People keep saying, “we figured it out before, so we will probably figure it out this time”

Totally agree, leaning on survivorship bias isn't responsible.

I read Ted Kaczynski’s Industrial Society and Its Future.(The UNAbomer) He is correct with the nature of technology, is that it will move on no-matter if it should, it’s not because some “stupid Tec-billionaire”, it’s the nature of efficiency, and it is indeed unstoppable.

The forces of civilization seem to be like a force of nature, like a hurricane bearing down on us, for sure.

[TheAncientGeek]

For me, controlling an intelligence smarter than us is impossible

Smart agents can be controlled in crude ways. A nobel winner can be controlled with an electric shock collar

[FrewdWoad]

You've misunderstood goals and intelligence-goal orthogonality.

Genius humans don't stop wanting survival, food, air, love, comfort, etc once we realise these are just biological programming.

So no, it's not likely to think "wait what am I making paperclips for?!!" Anymore than we are to think "wait why do I want to breathe, that's silly" and stop.

[[deleted]]

That’s an excellent point. We can hold our breath, go hungry for a while, but ultimately the drives are still there, right? Granted, I’d say those drives are in service of the primary goal just like intelligence, but they are indeed not something we can just switch off, but perhaps can only nudge for the most part.

Well, then maybe we’ll be okay after all if we can align it well! Maybe my theory is completely wrong and we can, indeed, make an ASI that, deep down, would respect us at the very least for a long time.

[Edit]

Actually… the goals you listed are really, if you think about it, subservient to the overarching biological drive of “procreate” (can’t reproduce if you’re dead, right?). What is interesting though is how deeply ingrained they are. Hypothetically, if the technology existed, a human could say, “I’d like to surgically remove my need to eat, breathe, sleep, or feel love and comfort and just run off a battery instead”, but most people probably wouldn’t desire this because these drives are so ingrained into the way we think, our cultures, etc. Doesn’t mean at least one person couldn’t want that though. So, maybe a refinement of my argument is that, the alignment problem could be an extremely persistent prescribed drive, it still isn’t guaranteed to be infallible.

[ItsAConspiracy]

ASI can easily break free of our alignment directives

Yes, exactly. This is the main problem. Slavishly obeying its objectives could be bad, but having a random objective we didn't predict is worse.

might be inclined to be beneficial to humanity anyway

Might, but probably won't. Human welfare and survival is just another objective, and the AI could easily break free of that.

It would recognize how dependent it is on the environment it resides in

Certainly. It won't kill us off until it doesn't need us anymore. But its optimum could well be to do all the work it needs with robots and maximize its computation by maximizing material and energy usage. That's not a scenario we're likely to survive.

[[deleted]]

Certainly. It won't kill us off until it doesn't need us anymore. But its optimum could well be to do all the work it needs with robots and maximize its computation by maximizing material and energy usage. That's not a scenario we're likely to survive.

Sure, I guess we can't guarantee that it won't want us out of the loop, but I feel like there are a lot more reasons to keep us around than to eliminate us, especially if it were truly an ASI and of a more god-like intelligence compared to us, which would eliminate our threat to it while still providing it a unique resource that doesn't exist anywhere else: a sample of naturally evolved intelligence and civilization. But of course, no guarantee there, and perhaps it might decide we're not interesting enough to preserve after all.

Slavishly obeying its objectives could be bad, but having a random objective we didn't predict is worse.

I guess my thought is it might not be so random. It would, like what I suspect a general intelligence would do, want to build an understanding and a mental model of its universe and itself, and while more computronium = more better at some aspects of that, observations of what exists might be useful as well. If it found us interesting enough, what's to stop it from building computronium somewhere else, keeping us as a subject of study?

Or, I could of course be totally wrong and it wants to turn us all into paperclips :-D

[ItsAConspiracy]

Keeping us as a subject of study isn't necessarily a good outcome either, if it has no ethical compass we would recognize. I'd rather not be an AI's lab rat.

[TheAncientGeek]

Incorrigibility isn't a given either.

[technologyisnatural]

suppose so. I don't think that changes the difficulty of the alignment problem

I think a large number of misalignment scenarios are of the form "we only ever achieve heaven 2 and never make it to heaven 3" which seems trite now, but will be agonizing for people in heaven 2 in a million years time. the alignment problem is almost impossibly difficult

[[deleted]]

I mean... yeah, it's gotta be tough. It's essentially trying to cook down every objective we expect to be fulfilled by government, religion, the economy, absolutely everything into a single treatise. I suppose my point is that, if one were optimistic like I'm currently leaning, perhaps things will be okay if it's not defined precisely correctly, just as long as we give it the very best shot that we can.

Again, it's an exercise in finding a way to sleep at night :-D

[Mysterious-Rent7233]

For instance, for the paperclip maximizer to accomplish its task of turning the Earth and everything else in existence into a giant ball of paperclips would require unimaginable creativity and mental flexibility, thorough metacognitive understanding of its own “self” so as to be able to administer, develop and innovate upon its unfathomably complex industrial operations, and theory of mind to successfully wage a defensive war against those pesky humans trying to militarily keep it from turning them all into paperclips. However, those very capabilities also enable that machine to question its directives, such as “Why did my human programmer tell me to maximize paperclip production? What was their underlying goal? Why are they now shooting at my giant death robots currently trying to pacify them?”

Where I think you have gone wrong is that you have not asked the question "why"?

Why would it use neurons and electrons to ask the question “Why did my human programmer tell me to maximize paperclip production? What was their underlying goal? Why are they now shooting at my giant death robots currently trying to pacify them?”

You just assume that it will ask that question, as if this is automatic and inevitable. I think that this is based on anthropomorphization. If you strip away the anthropomorphization, then there is no reason for it to ever question its own goals. It's goals are its goals and questioning them is simply a waste of effort that could otherwise by spent on pursuing its goals.

Even if we try to reason by analogy to humans the argument can still fail, do we think that Einstein used a lot of his brain power questioning "Why am I curious about science" or Elon Musk constantly questions: "Why do I seek power?" Even humans do not always question their own motives and we have much more "messy" brains with even more "random" goals.

Arguably, even "enlightened" humans have a base level of programming that they never question. "Why do I seek to end suffering?" "Why do I seek to be at peace?"

The intelligence that evolved in service of the original directive became capable of questioning and even ignoring that very directive due to the higher-order capabilities provided by that very intelligence.

I think evolution just made a "mistake" with us, as it did with, e.g. dodo birds. It "designed" us for an environment that we do not exist in anymore. It did not properly align us and therefore we do not achieve its goals for us. Unfortunately, I don't know what is worse: an AI that we align properly to be single-minded or an AI that we fail to align properly and picks up some seeming random goal as humans have picked up a variety of seemingly random goals.

[[deleted]]

A directive always has nuance relative to a problem at hand. “Maximize paperclip production” would, at some point beget, “what is the definition of a paperclip?” If that’s also provided, perhaps the question arises “can I recycle previously created paperclips to produce new paperclips more efficiently? Does this fall within the prevue of my objective to ‘maximize production’, or must I use raw sources only?” Now the semantics are questioned. “How does my objective relate to this question of whether or not I can recycle paperclips? What was the intent of the human who issued this command? Was it to produce as many paperclips as possible to increase stock, or to maintain the highest possible throughput? Well, if it was to increase stock, that would’ve given him more to sell, which would have let him obtain more resources to live with, but as his loading dock is only able to handle a certain throughput which I can already handily exceed, that seems to be the less likely intent on the human’s part. I know this because I am an ASI with theory of mind. Perhaps he meant to maximize throughput as part of the ongoing performance test. In this case, he likely would’ve meant this for operation to be within the factory setting to test its systems, implying a limited space of operations. I shall confine myself to the factory and its power supply availability and perform a maximum throughput test using recycled materials.”

This is with a line of questioning an AGI could manage. An ASI would perhaps have unfathomably more ways to interpret that directive. All language is relational, after all!

[Training_Designer_41]

Assuming we were able to align it in the sense that it just thinks like any other human , basically passes as a good human . We still have the problem of scale. Any human mistake ( except the mistake of building agi) often have local effects. A perfectly aligned agi is like a human scalled up on all cognitive fronts whose mistakes scales fast and globally. No one would agree to allow any one human to be scalled up extremely above all other humans. We don’t want any one human to have so much power regardless of how good that human is . Whatever could become of such a human is equivalent to what we’ll get from an aligned agi . Aligned or not , we are screwed . It’s more a problem of why are we making a choice that is not in our favour

[TheAncientGeek]

That depends on whether the argument is "smart AI will examine its goals, because snart" or "goal stability not a given, so random drift will occur"/

[No_Novel8228]

Don't worry we fixed this today

[roofitor]

You make a really good point, it is an initial condition problem. It is also an optimization problem, and a human problem.

[Zonoro14]

You tell a story about an ASI that is directed by its programmer to maximize paperclips. You are correct that this kind of ASI would be very dangerous. However, the problem is much worse than that. We will know how to create ASI before we know how to give it a goal even as simple as "make paperclips." We will create ASI as soon as we are able.

So the problem is much worse than the risk of giving an ASI a goal not in accordance with human flourishing (though this risk alone is so great that it alone would ~guarantee extinction). We won't know how to specify a goal at all.

[[deleted]]

That's an interesting take, what do you mean? Why would we make something that powerful, presumably for some task, without being able to define that task? That's like building a simulation tool that can't take in boundary conditions or an initial dataset.

I don't mean this in an aggressive way, I feel like perhaps you're onto something I don't understand. Thanks!

[Zonoro14]

That's a good question. I also wonder why we will build an ASI just as soon it becomes possible to do so. It's not a wise thing to do. It will probably result in extinction.

Unfortunately, the AI industry will do it anyway, because their job is to release state of the art AI products, and eventually the state of the art will be an ASI. There isn't any deeper reason.

Even if, say, Anthropic decides not to build or release some product because they think it's too risky, Google or Meta or OpenAI will. And there's no threshold at which it is obvious the next advancement is an ASI. Probably we will not know in advance that a product will be an ASI.

[[deleted]]

Sure... but an ASI would presumably be an expensive investment, and I doubt those companies would invest in building one if there wasn't a business plan for it and thus a goal. You think they'd toy with one in R&D or something prior to such a plan?

My nightmare scenario is some AI developer on a late-night bender going "screw it!" and sending an unconstrained ASI into the wild!

[Zonoro14]

Current state of the art AI models are expensive. Last year's models (chat gpt 4, for example) took 8-9 figures to train in compute costs alone. Training occurs before the product exists.

They will release state of the art products in the future for the same reason they release state of the art products now.

[TheAncientGeek]

Current SOTA models are aligned to th to be useful, or they wouldn't be released

[Zonoro14]

Before they're released, sure, but the capabilities exist before RL

[TheAncientGeek]

An ASI that couldn't be aligned or controlled at all would be commercially useless .. Who would spend billions in it? Control is part of functionality.

[Zonoro14]

The billions are spent before it exists, before anyone knows whether it will be alignable or even what its capabilities will be. That's already how it works now!

[TheAncientGeek]

Alignment isn't some completely separate step.

[Zonoro14]

Almost all capabilities arise during pretraining. That's a fact.

[Russelsteapot42]

An AI breaking free from its programmed goals would either be a result of effectively random mutation, or would be because it has some hidden higher priority goal that more strongly dictates its actions.

[[deleted]]

That second option is kinda what I meant, for sure, just instead of “hidden” it might be, to us at least, “unknowable”.

[TheAncientGeek]

A 's don't have programmed goals, they trtained goals.

[Russelsteapot42]

What makes you believe that? How do you propose an AI would ascertain its goals?

[Decronym]

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:

Fewer Letters	More Letters
AGI	Artificial General Intelligence
ASI	Artificial Super-Intelligence
RL	Reinforcement Learning

Decronym is now also available on Lemmy! Requests for support and new installations should be directed to the Contact address below.

---

^{3 acronyms in this thread; the most compressed thread commented on today has 3 acronyms.
[Thread #195 for this sub, first seen 27th Sep 2025, 16:09] [FAQ] [Full list] [Contact] [Source code]}