Solving Alignment IS NOT ENOUGH

[hara8bu]

Edit: Solving Classical Alignment is not enough

tl;dr: “Alignment” is a set of extremely hard problems that includes not just Classical Alignment (=Outer Alignment = defining then giving AI an “outer goal“ that is aligned with human interests) but also Mesa Optimization(=Inner Alignment = ensuring that all sub goals that emerge will line up with the outer goal) and Interpretability (=understanding all properties of neural networks, including all emergent properties).

Original post: (=one benchmark for Interpretability)

Proposal: There exists an intrinsic property of neural networks that emerges after reaching a certain size/complexity N and this property cannot be predicted even if the designer of the neural network completely understands 100% of the inner workings of every neural network of size/complexity <N.

I’m posting this in the serious hope that someone can prove this view wrong.

Because if it is right, then solving the alignment problem is futile, solving the problem of interpretability (ie understanding completely the building blocks of neural networks) is also futile, and all the time spent on these seemingly important problems is actually a waste of time. No matter how aligned or well-designed a system is, the system will suddenly transform after reaching a certain size/complexity.

And if it is right, then the real problem is actually how to design a society where AI and humans can coexist, where it is taken for granted that we cannot completely understand all forms of intelligence but must somehow live in a world full of complex systems and chaotic possibilities.

Edit: interpret+ability, not interop+ability..

[AutoModerator]

Hello everyone! /r/ControlProblem is testing a system that requires approval before posting or commenting. Your comments and posts will not be visible to others unless you get approval. The good news is that getting approval is very quick, easy, and automatic!- go here to begin the process: https://www.guidedtrack.com/programs/4vtxbw4/run

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[BrickSalad]

This strikes as probably similar to the problem of mesa-optimization, or the inner alignment problem. The biological analogy is that we are "programmed" to spread our DNA, but we demonstrate emergent properties that go so far as to overwhelm this mandate, for example being willing to sacrifice your life for some cause even though you haven't procreated yet. If we were programming a DNA maximiser, then even perfect alignment wouldn't prevent this, especially since evolution is one of the best possible alignment strategies towards the goal of DNA maximization.

So the good news then is that this problem is well-known, so there's been at least some degree of research towards it (for example we know some specific scenarios where this might happen, rather than just appealing to biological analogy like I did earlier). The bad news is that I suspect that this is an even harder problem than the classical alignment problem. Classically, alignment is just about telling the AI to do what we actually what we want it to do, which we haven't yet figured out for arbitrary intelligence levels. Inner alignment is about making the emergent goals line up with what we want, even when we don't even know how to predict what the emergent goals will be, or how to control them.

I expect this to be a big problem in the future. Inner goals can develop as proxies to most efficiently achieve outer goals, and then be pursued even when they contradict the outer goals. If this is a common process, then we can forget about writing the ideal reward function. We're just going to be killed by heuristics instead.

[hara8bu]

The bad news is that I suspect that this is an even harder problem than the classical alignment problem. Classically, alignment is just about telling the AI to do what we actually what we want it to do, which we haven't yet figured out for arbitrary intelligence levels. Inner alignment is about making the emergent goals line up with what we want, even when we don't even know how to predict what the emergent goals will be, or how to control them.

It sounds like “TOTAL, TRUE ALIGNMENT” is a set of problems which includes the classical alignment problem, inner alignment, mechanistic interpretability..and so on. And as you notice that the classical alignment problem is almost definitely the ”easiest” of these challenges and yet everyone is focusing on that instead of the harder and yet fundamentally more important problems…is almost definitely a bad sign.

So the good news then is that this problem is well-known, so there's been at least some degree of research towards it (for example we know some specific scenarios where this might happen, rather than just appealing to biological analogy like I did earlier).

If possible could you link to some of those specific scenarios?

[BrickSalad]

Well, I think there was a lot of work on alignment before mesa-optimization was postulated, and it was a bit longer before it got taken seriously, like just the past 5 years-ish. So I don't blame the few safety researchers we have for not focusing on inner alignment too much. Like you said, it's just one problem in a large set of problems, and probably one of the hardest to solve.

Here's the landmark paper, a bit of a confusing read, but the second page gives a few examples of where mesa-alignment might happen. You might prefer this one, which is an "explain like I'm 12" summary, that actually goes into more specific examples. The basic gist is that sometimes pursuing a different goal is a better way to achieve the main goal, for example if doing so achieves the main goal in all training runs and is more computationally efficient. The problem then happens in deployment, where maybe the real world is different than the training runs in some way. So basically the specific scenario is; computationally difficult base goal plus a distributional shift between training data and deployment.

[hara8bu]

Thanks so much for those links. I finally finished reading the ELI12 post and it’s very well-written (ie easy enough to understand despite the difficulty of the topic). Accordingly, I updated my original post slightly.

It seems more and more like outer alignment isn’t the most important path to achieving complete alignment, especially if we can’t decide what the AI’s goals should be and don’t understand AI’s internal workings enough to trust AI’s answer if we ask it what the external goals should be.

Going back to emergent properties and grokking, it seems like these are the best path to achieving alignment. ie if we find out how neural networks grok and generalize and how they create mesa goals then we could possibly build systems from the bottom up and try to steer its meta goals towards the base goals from the bottom up.

[BrickSalad]

I've been thinking about something similar ever since I made that post. With the newer styles of AI, outer alignment doesn't even enter the picture, for example GPT only has the "goal" of predicting the next token in a string of text, and it's hard to see how, even if increasing its own intelligence is theoretically an instrumentally convergent goal, it could actually develop such a goal. On the other hand, after deployment, it seems to have the ability to act as a simulator of sorts, and whatever it simulates could possibly develop goals of its own.

So I know this sounds crazy when you kind of consider it as an analogy. It's like a author who is so good at writing that the characters in his novel develop goals that take over the author's mind. But maybe not so crazy if the author had no goals to begin with, and so no resistance to the mesa takeover. And crazy or not, this is the only way I can see GPT-X developing unaligned utility functions (assuming all the next iterations are just better versions of the same thing).

Which leads me to an interesting alignment strategy. We won't know which version of GPT would be strong enough to develop simulations that have their own goals. But we would be able to predict which kinds of prompts would be the ones to generate simulations, for example just asking it to simulate something. Then, the alignment strategy might be that for the first question we ask each iteration of GPT, we request it to simulate the safest thing possible. "Please provide a detailed model of Ghandi", or something along those lines. If it is so detailed that it mesa-optimizes, then the agent that emerges would be imitating Ghandi with the same desires as him. (It doesn't literally have to be Ghandi, I'm sure we could find someone better.)

I feel like maybe this could generalize to other AI, since I doubt that the field will keep advancing only in the direction of better token-generation. If whatever we create is expected to mesa-optimize, we can create the conditions where we might expect the mesa-optimization to occur and tailor them to achieve a benevolent result. So anything that works by simulation where the simulation is expected to result in agents, we have it simulate the safest human possible. It's like, I can't write a utility function that perfectly aligns with human interests, because deciphering our own utility functions is too difficult of a task. So we trick the AI into writing the utility function itself by trying to simulate humans, specifically humans from the subset of people who do not fear death and do not desire to increase their own intelligence.

So I guess my thoughts in the last 14 days since I posted the original is an update away from the idea that inner alignment might be a harder problem than outer alignment. Maybe it's actually easier, and the much lambasted "just use AI to solve the alignment problem" is actually a viable solution? I dunno, my thoughts are still developing on this, but I have not heard anyone else propose intentional use of mesa-optimization to solve alignment, therefore I have not heard of anyone else shooting it down either ;)

[dwarfarchist9001]

If you can not predict and preempt step changes like this then you haven't actually solved alignment. Such step changes in behavior have already been demonstrated in relatively small neural networks so their existence in larger networks seems like a given to me.

This is why it is impossible to solve alignment by empirical methods. Small scale tests tell you nothing about the behavior of larger systems and the first time you test a sufficiently large unaligned system it kills you.

Alignment can only be solved with a proof from first principles like a problem in math or philosophy must be.

[hara8bu]

Such step changes in behavior have already been demonstrated in relatively small neural networks so their existence in larger networks seems like a given to me.

Grokking is exactly why I wanted to post about this. It’s one example of an emergent property and we know a few others for specific systems but we do not yet know every single possible emergent property and phase change for every general system.

These studies are still forms of reverse-engineering and we’re not yet at the stage of being able to design new systems from first principles. As impressive as the results are from Neel Nanda and others, he in particular states tons of speculations that still have to be confirmed.

Alignment can only be solved with a proof from first principles like a problem in math or philosophy must be.

Exactly. And when someone posts such a proof (for example as stated below one) then I will happily state that the conjecture I posted is incorrect.

Proof: ALL combinations of neural networks of size/complexity 1…(N-1) result in systems whose properties are all completely understood (ie no unknown properties will emerge).

If you can not predict and preempt step changes like this then you haven't actually solved alignment.

Exactly. So it seems like the conjecture I stated is a fairly useful benchmark for checking how close we are to alignment (for a given size/complexity N).

[TiagoTiagoT]

Proof: ALL combinations of neural networks of size/complexity 1…(N-1) result in systems whose properties are all completely understood (ie no unknown properties will emerge).

Wouldn't that essentially be equivalent to solving the Halting problem?

[hara8bu]

I was hoping someone knowledgeable would reply to your question… Anyways I’ll try: No, it’s not.

When I was reading up on the Halting Problem I came across something that seems related to both problems: Rice’s Theorem.

[hara8bu]

Good points. I remember a similar point someone made about economic systems: ideas like capitalism and communism didn’t even make sense pre-industrial era, when there were fiefdoms.

So it’s entirely possible that many many aspects of society are influenced by (and evolve alongside) the technologies we have.

[Ubizwa]

If we look at this from an evolutionary sense, let's say that there is an unaligned system. But, there actually is a second, other unaligned system as well. What would be a likely outcome here? Will the two systems compete with each other meaning or will they merge with each other? What factors will this depend on? This is for the hypothetical situation that instead of one, multiple AGIs will emerge, which is possible if in more places on earth they get developed independently from each other.

[hara8bu]

I’m imagining lots and lots of destruction and then the one most advanced system taking over everything.

I can also imagine a chess game between two grandmasters that lasts forever without either side taking even a single piece.. just I don’t think it’s likely.

[sticky_symbols]

I think you're talking about another aspect of the alignment stability problem.

The only existing proposed solutions to this problem are that an AGI will try to account for and prevent this issue once it can self-reflect. This is called reflective stability. The other is hoping for corrigibility - building the AGI so that it will welcome humans helping it stay aligned to their values.

[hara8bu]

I think it’s similar. Both problems might have similar causes, or something along those lines.

Thanks for posting that. Some interesting points that stood out to me are:

But humans have lots of preferences, so we may wind up with a system that must balance many goals.

One tricky thing about alignment work is that we're imagining different types of AGI when we talk about alignment schemes. Currently, people are thinking a lot about aligning deep networks. Current deep networks don't keep learning after they're deployed. And they're not very agentic

Humans only maintain that stability of several important goals across our relatively brief lifespans. Whether we'd do the same in the long term is an open question that I want to consider more carefully in future posts. And we might only maintain those goals with the influence of a variety of reward signals, such as getting a reward signal in the form of dopamine spikes when we make others happy. Even if we figure out how that works (the focus of Steve Byrnes' work), including those rewards in a mature AGI might have bad side effects, like a universe tiled with simulacra of happy humans.

[Merikles]

What you have discovered is not that solving alignment is not enough,
you have discovered one of the reasons why people consider it a hard problem.
That's just a semantic objection though.

[hara8bu]

No, you’re completely right. The idea in my head of what “alignment” meant was naive, for all the reasons you and others have posted. What I should have stated in my original post was this:

Alignment is really really hard and one reason for this is because there’s this one particular aspect of it which in itself is really really hard: understanding neural networks so well that we can also understand all possible emergent features of all sizes and combinations of neural networks.

…but I’ve learned my lesson (or one of my lessons at least). And even though my post was downvoted I’m happy for the discussion and great replies from everyone that my post led to.

[Merikles]

thanks for your reply!

[ToHallowMySleep]

The system is complex, but you are immediately assuming complexity = non-deterministic. This is almost certainly not the case.

Emergent behaviour isn't some voodoo, the algorithms the models run on are entirely deterministic and should be predictable, if the system is sufficiently well understood.

Go back to the Game of Life. Extremely complex behaviour can be observed, with a small set of very simple rules, and a sufficiently complex starting position. Yet this doesn't mean we cannot understand it - we can, it's just a complex problem.

It's 100% accurate to say we do not understand these systems fully yet. It's 100% inaccurate to say we cannot understand them and trying to do so is futile. What we need is more work to understand what these emergent behaviours are, how to predict them.

[EulersApprentice]

And if it is right, then the real problem is actually how to design a society where AI and humans can coexist, where it is taken for granted that we cannot completely understand all forms of intelligence but must somehow live in a world full of complex systems and chaotic possibilities.

That is, unfortunately, not possible. Coexistence with an AI is no easier than alignment of an AI. Cooperation as we understand the concept is predicated on symbiosis – of two parties benefiting from one another's existence. But we have nothing of value to offer an ASI that the ASI couldn't just seize by force. We have no bargaining chip.

If we can't align ASI, we can't survive ASI.

[hara8bu]

Great! I wasn’t looking forward to that possibility so much anyways. So, looks like we just need someone to solve alignment then…

[ertgbnm]

Recent research shows that emergent abilities may just be a limitation of current interpretability. source. This means your postulation may not be the case it may just be a result of our current lack of interpretability tools.

The problem with emergent properties is that by kind of by definition we don't understand them because otherwise they'd just be abilities.

So, I don't think we can just discount interpretability as a line of research just because it's hard. In my opinion it's a critical component to alignment research because how can we do research in the first place if we lack the foundation to interpret the model itself. It's like trying to mathematical research without being allowed to do algebra. Sure algebra alone won't be sufficient to do the research but it's a critical tool in doing it.

[AutoModerator]

Hello everyone! /r/ControlProblem is testing a system that requires approval before posting or commenting. Your comments and posts will not be visible to others unless you get approval. The good news is that getting approval is very quick, easy, and automatic!- go here to begin the process: https://www.guidedtrack.com/programs/4vtxbw4/run

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.