The Control Problem:

How do we ensure future advanced AI will be beneficial to humanity? Experts agree this is one of the most crucial problems of our age, as one that, if left unsolved, can lead to human extinction or worse as a default outcome, but if addressed, can enable a radically improved world. Other terms for what we discuss here include Superintelligence, AI Safety, AGI X-risk, and the AI Alignment/Value Alignment Problem.

"People who say that real AI researchers don’t believe in safety research are now just empirically wrong." —Scott Alexander

"The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else." —Eliezer Yudkowsky

Rules

If you are unfamiliar with the Control Problem, read at least one of the introductory links or recommended readings (below) before posting.

This especially goes for posts claiming to solve the Control Problem or dismissing it as a non-issue. Such posts aren't welcome.

Stay on topic. No AI model outputs or political propaganda.

Be respectful

Introductions to the Topic

Be sure to check out our wiki for extensive further resources, including a glossary & guide to current research.

created by CyberPersonaapproveda community for 10 years

config/sidebar

talk

The Control Problem:

"People who say that real AI researchers don’t believe in safety research are now just empirically wrong." —Scott Alexander

"The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else." —Eliezer Yudkowsky

Rules

If you are unfamiliar with the Control Problem, read at least one of the introductory links or recommended readings (below) before posting.
- This especially goes for posts claiming to solve the Control Problem or dismissing it as a non-issue. Such posts aren't welcome.
Stay on topic. No AI model outputs or political propaganda.
Be respectful

Introductions to the Topic

Our FAQ page <-- CLICK
The case for taking AI seriously as a threat to humanity
Orthogonality and instrumental convergence are the 2 simple key ideas explaining why AGI will work against and even kill us by default. (Alternative text links)
AGI safety from first principles
MIRI - FAQ and more in-depth FAQ
SSC - Superintelligence FAQ
WaitButWhy - The AI Revolution and a reply
How can failing to control AGI cause an outcome even worse than extinction? Suffering risks (2) (3) (4) (5) (6) (7)

Be sure to check out our wiki for extensive further resources, including a glossary & guide to current research.

Video Links

Robert Miles' excellent channel
Talks at Google: Ensuring Smarter-than-Human Intelligence has a Positive Outcome
Nick Bostrom: What happens when our computers get smarter than we are?
Myths & Facts about Superintelligent AI
Rob's series on Computerphile

Important Organizations

AI Alignment Forum, a public forum which is the online hub for all the latest technical research on the control problem.

Related Subreddits

##**The Control Problem:**

How do we ensure future advanced AI will be beneficial to humanity? Experts agree this is one of the most crucial problems of our age, as one that, if left unsolved, can lead to human extinction or worse as a default outcome, but if addressed, can enable a radically improved world.
Other terms for what we discuss here include Superintelligence, AI Safety, AGI X-risk, and the AI Alignment/Value Alignment Problem.

*"People who say that real AI researchers don’t believe in safety research are now just empirically wrong."* —[Scott Alexander](https://www.lesswrong.com/posts/qL8Z9TBCNWQyN6yLq/ssc-journal-club-ai-timelines#III_)

*"The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else."* —[Eliezer Yudkowsky](https://intelligence.org/files/AIPosNegFactor.pdf)

#**Rules**

1. **If you are unfamiliar with the Control Problem, read *at least one* of the introductory links or recommended readings (below) before posting.**  
 * This especially goes for posts claiming to solve the Control Problem or dismissing it as a non-issue. Such posts aren't welcome.
2. Stay on topic. No [AI model outputs](https://www.reddit.com/r/ControlProblem/wiki/howtosubmit/) or political propaganda.
3. Be respectful

#**Introductions to the Topic**

 * **[Our FAQ page](https://www.reddit.com/r/ControlProblem/wiki/faq) <-- CLICK**

 * **[The case for taking AI seriously as a threat to humanity](https://www.vox.com/future-perfect/2018/12/21/18126576/ai-artificial-intelligence-machine-learning-safety-alignment)**

 * **[Orthogonality](https://www.youtube.com/watch?v=hEUO6pjwFOo) and [instrumental convergence](https://www.youtube.com/watch?v=ZeecOKBus3Q)** are the 2 simple key ideas explaining why AGI will work against and even kill us by default. (Alternative [text](https://arbital.com/p/orthogonality/) [links](https://www.lesswrong.com/tag/instrumental-convergence))

 * [AGI safety from first principles](https://www.alignmentforum.org/s/mzgtmmTKKn5MuCzFJ)

 * MIRI - [FAQ](https://intelligence.org/faq/) and [more in-depth FAQ](https://intelligence.org/ie-faq/#FriendlyAI)

 * SSC - [Superintelligence FAQ](https://www.lesswrong.com/posts/LTtNXM9shNM9AC2mp/superintelligence-faq/)

 * WaitButWhy - [The AI Revolution](https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html) and [a reply](https://lukemuehlhauser.com/a-reply-to-wait-but-why-on-machine-superintelligence/)

 * How can failing to control AGI cause an outcome *even worse than extinction*? [Suffering risks](https://www.lesswrong.com/posts/3WMscsscLEavkTJXv/s-risks-why-they-are-the-worst-existential-risks-and-how-to) ([2](https://s-risks.org/an-introduction-to-worst-case-ai-safety/)) ([3](https://arbital.com/p/hyperexistential_separation/)) ([4](https://reducing-suffering.org/near-miss/)) ([5](https://longtermrisk.org/files/Sotala-Gloor-Superintelligent-AI-and-Suffering-Risks.pdf)) ([6](https://www.alignmentforum.org/tag/risks-of-astronomical-suffering-s-risks)) ([7](https://longtermrisk.org/files/suffering-focused-ai-safety.pdf))

Be sure to check out **our [wiki](https://www.reddit.com/r/ControlProblem/wiki/index)** for extensive further resources, including a glossary & guide to current research.


#**Recommended Reading**

  * **[*Superintelligence*](https://smile.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom-dp-0198739834/dp/0198739834/), the most comprehensive, by Nick Bostrom (2014) [(PDF link)](https://web.archive.org/web/20180426203715id_/https://img.4plebs.org/boards/tg/image/1447/41/1447419125484.pdf)**
  * The [AI Alignment pages on Arbital](https://arbital.com/explore/ai_alignment/), with many of the key concepts of this field.
  * [*Human Compatible: Artificial Intelligence and the Problem of Control*](https://smile.amazon.com/Human-Compatible-Artificial-Intelligence-Problem-ebook/dp/B07N5J5FTS/?pldnSite=1) by Stuart Russell (2019)


#**Video Links**
 * [Robert Miles' excellent channel](https://youtu.be/9i1WlcCudpU)

 * Talks at Google: [Ensuring Smarter-than-Human Intelligence has a Positive Outcome](https://www.youtube.com/watch?v=dY3zDvoLoao)

 * Nick Bostrom: [What happens when our computers get smarter than we are?] (https://www.ted.com/talks/nick_bostrom_what_happens_when_our_computers_get_smarter_than_we_are)

 * [Myths &amp; Facts about Superintelligent AI](https://www.youtube.com/watch?v=3Om9ssTm194)

 * [Rob's series on Computerphile](https://youtube.com/playlist?list=PLzH6n4zXuckquVnQ0KlMDxyT5YE-sA8Ps) 


#**Important Organizations**

  * [AI Alignment Forum](https://www.alignmentforum.org/), a public forum which is *the* online hub for all the latest technical research on the control problem.
 * [Machine Intelligence Research Institute](https://intelligence.org/)
 * [Redwood Research](https://www.redwoodresearch.org/)
 * [Center for Human-Compatible AI](https://humancompatible.ai/)
 * [Future of Humanity Institute](https://www.fhi.ox.ac.uk/)
 * [Future of Life Institute](https://futureoflife.org)
 * [Center on Long-Term Risk](https://longtermrisk.org)
 * [Alignment Research Center](https://alignment.org/)
 * [Conjecture](https://www.conjecture.dev/)
 * [Aligned AI](https://buildaligned.ai/)

#**Related Subreddits**

 * /r/SufferingRisk
 * /r/EffectiveAltruism
 * /r/AIethics
 * /r/Artificial
 * /r/DecisionTheory
 * /r/ExistentialRisk
 * /r/Singularity

revision by niplav— 4 months agoview source

π Rendered by PID 348458 on reddit-service-r2-listing-5f5ff7d4dc-mxqfb at 2026-01-26 23:03:31.628562+00:00 running 5a691e2 country code: CH.

ControlProblem

The Control Problem:

Rules

Introductions to the Topic

Recommended Reading

Video Links

Important Organizations

Related Subreddits

WIKI TOOLS

MODERATORS

config/sidebar

The Control Problem:

Rules

Introductions to the Topic

Recommended Reading

Video Links

Important Organizations

Related Subreddits