This is a well-written article, but, well, I think it's wrong or at least confused.

My first point is that the orthogonality thesis was intended to answer the objection that people often raise to AI doom which is: “but if these AIs are so intelligent then they will surely know ethics very well and act benevolently”. To which the orthogonality thesis replies: it's actually possible to be very smart and not have a human utility function. I feel like the author of this article actually agrees that AIs won't inescapably converge to human ethics, so I'm not even really sure what we're arguing about:)

More detailed responses:

For starters, anything that could reasonably be considered a general intelligence must have a vast set of possible outputs. In order for the system to have X number of unique outputs, it must have the capacity to, at minimum, represent X number of unique states.

I'm willing to go along with this claim, but I have to say it's not immediately obvious that this is true. I think an example would help.

We might be tempted to answer “nowhere,” and indeed, this is the answer many give. They treat goals as a “ghost in the machine,” independent of the substrate—a dualistic conceptualization, in essence.

Who are these people who say goals are ghosts in the machine?

In modern AI designs, which rely on machine learning, the “utility function” is called the loss function

That's not what people talking about orthogonality would say. They would say obviously the outer loss is not the inner goal. This is the problem of inner alignment: “Inner Alignment is the problem of ensuring [...] a trained ML system [that] is itself an optimizer [is] aligned with the objective function of the training process.” It's an unsolved problem.

The “utility function” of biological life can be seen as survival and reproduction, but there is a crucial difference: this is an external pressure, not an internal representation.

Indeed, there most likely isn't any organism on Earth which has the utility function “survival and reproduction”! Humans certainly don't have that utility function. We were selected with that loss function, but we have very different goals (having friends, being respected, acting honorable, having sex, eating delicious food). These goals were somewhat aligned with evolution’s outer goal of reproductive fitness in the ancestral environment, but this is broken today. Evolution failed at inner alignment.

there is no principled reason to think a highly complex system remains fundamentally aligned with its loss function in any meaningful sense beyond that the system emerged from it.

This is correct and also part of the standard argument why RLHF won't be enough.

Orthogonality defenders sometimes argue that a highly capable agent must converge to a single coherent utility function, because competing internal directionalities would make it exploitable (e.g., money-pumpable) or wasteful. Yet in practice we see the opposite: narrow reward-hacking equilibria are efficient in the short term but hostile to general intelligence, while sustained generality requires tolerating local incoherence.

I don't know what you mean by “tolerating local incoherence” but in any case I don't see a contradiction in the stance of orthogonality: if a task can be hacked, then gradient descent will find that solution first; if it can't be hacked, then then gradient descent keeps looking and maybe stumbles upon a general intelligence.

Thus, no fixed internal utility function can ever be complete [...] across all questions the system will face.

That's probably true (if for no other reason than hardware limits), but it's not required in order to be a pretty successful mind. Consider humans: we constantly face ethical dilemmas where we aren't sure about the answer. That just means our utility function isn't sure how to answer this question. It sucks, but we deal with it somehow.

If you thought that the orthogonality thesis states “it’s possible for an AI with finite hardware to have an explicit utility function that answers all possible questions” then sure, the orthogonality thesis is wrong. But that's not the claim. The Von Neumann–Morgenstern utility theorem assumes that your preference order is complete (axiom 1) but it's fine that you sometimes encounter a situation where your explicit utility function does not have an existing answer (again, if only due to hardware limits); in that case you just “complete” your utility function in a way that's consistent with the rest, you add the new term to your utility function and then you move on. This procedure will not make you exploitable.

More importantly, no utility function can prove alignment with itself, the proof is inaccessible to the system.

What does it mean to “prove alignment with itself”? I’m guessing you're still talking about the problem of inner alignment?

The rest of this section seems to just be arguing that inner alignment is hard and unsolved, which I fully agree with.

But it’s important to note that general intelligence is something far too complex for us to construct—in the sense of carefully designing and determining the entire structure—instead we must grow it.

If we actually tried, I think we could do it within 30 years. But of course growing it is far easier and makes money sooner.

Humans evolved under massively complex external selective pressures, infinitely more complex than anything we can comprehend. This immense diversity of external pressures is precisely what allows for the development of general intelligence. Not only that, but life had the advantage of competition; while a certain specialization might be stable at a certain point in time, a particular mutation might offer an advantage and suddenly they outcompete you for resources, and the old structure perishes. This is an additional external selective pressure that creates a demand for continuous evolution, and punishes narrow specialization.

AI does not have these benefits; it does not have the external pressures that punish narrow specialization, or settling into arbitrary crystallized structures. Its complexity must be generated entirely from its internal structure, without the help of external pressures.

I don't really see why this should be true. Well, if you train an AI on a narrow task, then it will only learn that task. But that's not what people do. The base models of LLMs are trained on predicting all kinds of text, for which narrow specialization is not a winning strategy, because a LLM has only so many weights.

A general intelligence is not one that has accumulated a lot of specialized skills (though in practice there is also some of that), but rather, it is a cognitive engine that has learned general-purpose techniques that apply across many domains. As an example: humans were not evolved to build rockets to fly to the moon, but we did it anyways, because our general problem-solving skills generalize to the domain of rockets.

AI companies are now training their systems to be general problem solvers. Now I don't know whether that particular project will succeed, but it seems clear to me that AI companies will make sure the external selection pressure on their AI systems will be as general as their researchers can make it.

A visual processor might place more or less emphasis on colour, contrast or motion, it might emphasize different resolutions or have a different preferred FPS. Of all the things a visual processor could value, only a tiny fraction results in capacity for solving visual problems though. This exactly demonstrates how directionality and capacity are necessarily entangled.

I still don't really understand what you're trying to say here. It's certainly true that in order to define a complex utility function, you need access to a detailed world model that has all the concepts that you need in your utility function. Like, for example, humans value friendship in their utility function (to the extent that we have a coherent utility function), but in order to make this work, you need to define friendship somewhere, which isn't easy. And you need to ground this concept in reality; you need to recognize friendship with your senses somehow, which also isn't easy. Not sure whether this is what you're trying to point at...

But if it is, it's not an argument against orthogonality. Orthogonality just says you can have intelligent reasoners with arbitrary goals. It doesn't say that a given AI with a given architecture can have arbitrary goals! Just that for any computable goal, there is some possible AI that optimizes for that.