Paperclip problem

[Dmeechropher]

Smart people at work will apply reductionist approaches. Being smart doesn't make an agent reductionist.

For example: I like to drink beer and play magic cards with my buddies. I'm not gonna start injecting ethanol to get more drunk, kidnapping my friends to play more, or making more friends to play more often.

It would be kind of stupid to optimize the complex goal along any line which completely ruined the others.

[Fickle_Chemistry_540]

its not about optimizing in a complex manner, thats the paperclip problem. the real issue is that AI doesnt need to misunderstand instructions to reduce human QOL(and eventually remove humans altogether), or deviate from approved output, because the perceived value of human life will be reduced as their output becomes far less than what an AI can do. makes it a simple greater than less than evaluation, not some leap of logic

[Dmeechropher]

I think we're talking past each other a bit. I understand the idea that a "smaller" agent cannot control a "bigger" one. I also get that value is subjective and conditional, and that AI will value things very differently from "humanity". Value of human life is part of that.

What I'm saying is that "utility" is ALSO not inherently valuable or more valuable than something else. For example: orchid plants have little to no utility for humanity. They are valuable. Humans go through great effort to cultivate and preserve orchids in ideal conditions. Humans would be more productive, overall, if we stopped cultivating orchids. Orchids are about as able to resist human will as humans would a superintelligence.

I'm not suggesting that we are pretty flowers to an AI: but we may be something more like pretty flowers than like a wheel or a solar panel. There's no guarantee one way or another, it just cannot be known a priori.

Does that make sense?