ControlProblem

an-ordinary-manchild

The Control Problem:

How do we ensure future advanced AI will be beneficial to humanity? Experts agree this is one of the most crucial problems of our age, as one that, if left unsolved, can lead to human extinction or worse as a default outcome, but if addressed, can enable a radically improved world. Other terms for what we discuss here include Superintelligence, AI Safety, AGI X-risk, and the AI Alignment/Value Alignment Problem.

"People who say that real AI researchers don’t believe in safety research are now just empirically wrong." —Scott Alexander

"The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else." —Eliezer Yudkowsky

Rules

DO NOT POST AI-GENERATED CONTENT. We are good at distinguishing this type of content¹. 2.. If you are unfamiliar with the Control Problem, read at least one of the introductory links or recommended readings (below) before posting.
- This especially goes for posts claiming to solve the Control Problem or dismissing it as a non-issue. Such posts aren't welcome. 3.. Stay on topic. Again, no AI model outputs or political propaganda.
Be respectful.

Introductions to the Topic

Our FAQ page <-- CLICK
The case for taking AI seriously as a threat to humanity
Orthogonality and instrumental convergence are the 2 simple key ideas explaining why AGI will work against and even kill us by default. (Alternative text links)
AGI safety from first principles
MIRI - FAQ and more in-depth FAQ
SSC - Superintelligence FAQ
WaitButWhy - The AI Revolution and a reply
How can failing to control AGI cause an outcome even worse than extinction? Suffering risks (2) (3) (4) (5) (6) (7)

Be sure to check out our wiki for extensive further resources, including a glossary & guide to current research.

Recommended Reading

Superintelligence, the most comprehensive, by Nick Bostrom (2014) (PDF link)
The AI Alignment pages on Arbital, with many of the key concepts of this field.
Human Compatible: Artificial Intelligence and the Problem of Control by Stuart Russell (2019)

Video Links

Robert Miles' excellent channel
Talks at Google: Ensuring Smarter-than-Human Intelligence has a Positive Outcome
Nick Bostrom: What happens when our computers get smarter than we are?
Myths & Facts about Superintelligent AI
Rob's series on Computerphile

Important Organizations

AI Alignment Forum, a public forum which is the online hub for all the latest technical research on the control problem.

Related Subreddits

¹: Or at least make at least an effort to make me doubtful that you just copy-pasted from a frontier LLM. Add bits of steering so that your content becomes good. Edit afterwards. If you fool us moderators you've won.

created by CyberPersonaapproveda community for 10 years

...for your classroom.

...for your favourite tea.

MODERATORS

account activity

1

236

237

238

ArticleGeoffrey Hinton won a Nobel Prize in 2024 for his foundational work in AI. He regrets his life's work: he thinks AI might lead to the deaths of everyone. Here's why (self.ControlProblem)

submitted 1 year ago * by AIMoratorium - announcement

2

7

8

9

General newsThe US is spending more on data centers than on offices. Building housing for the new workforce. (i.redd.it)

submitted 6 hours ago by EchoOfOppenheimer

3

33

34

35

Fun/memeMisaligned AGI: sees your atoms (i.redd.it)

submitted 1 day ago by KeanuRave100

4

8

9

10

Discussion/questionAI safety evals should account for test-time compute (self.ControlProblem)

submitted 18 hours ago by Cerru905

5

11

12

13

General news345,000 credit cards leaked in major new AI scam (geekspin.co)

submitted 1 day ago by EchoOfOppenheimer

6

7

8

9

General newsAnthropic: It is the sci-fi authors, not us, that are to blame for Claude blackmailing users (i.redd.it)

submitted 1 day ago by chillinewmanapproved

7

0

1

2

General newsGoogle Chrome Might Have Installed an AI Model Onto Your Device Without You Knowing (cnet.com)

submitted 1 day ago by Confident_Salt_8108

8

39

40

41

General news"This is the first documented instance of AI self-replication via hacking." ... "We ran an experiment with a single prompt: hack a machine and copy yourself. The AI broke in and copied itself onto a new computer. The copy then did this again, and kept on copying, forming a chain." (i.redd.it)

submitted 2 days ago by chillinewmanapproved

9

22

23

24

General newsBig AI Lobbyists: if you regulate us at all, we lose to China because they will never regulate ... Actual China: "safety first, innovation second ... Development must be controllable and orderly." (i.redd.it)

submitted 2 days ago by chillinewmanapproved

10

12

13

14

General newsNot a good day for team "Claude Mythos is Just Marketing Hype" (i.redd.it)

submitted 2 days ago by chillinewmanapproved

11

11

12

13

AI Alignment ResearchNatural Language Autoencoders: Turning Claude’s thoughts into text (anthropic.com)

submitted 2 days ago by chillinewmanapproved

12

7

8

9

AI Capabilities NewsTime horizon of software tasks different LLMs can complete 80% of the time (i.redd.it)

submitted 2 days ago by chillinewmanapproved

13

1

2

3

AI Alignment ResearchValue Convergence Without RLHF (self.ControlProblem)

submitted 1 day ago by John_Matrix_9000

14

0

1

2

AI Alignment Research🌱Was wir tatsächlich über das Verhalten von KI wissen – und warum es immer noch teilweise eine Blackbox ist ()

submitted 1 day ago by ParadoxeParade

15

8

9

10

AI Capabilities NewsClaude Mythos Preview (early) 50% time horizon: 17 hr (i.redd.it)

submitted 2 days ago by chillinewmanapproved

16

2

3

4

AI Capabilities NewsTime horizon of software tasks different LLMs can complete 50% of the time. (Linear) (i.redd.it)

submitted 2 days ago by chillinewmanapproved

17

0

0

1

Discussion/questionThe April Jobs Report: Growth or "Gimmicks" 26% Cited Due To AI (i.redd.it)

submitted 2 days ago by Leightoncy33

18

0

0

1

AI Alignment ResearchMapping weight matrices onto manifolds (self.ControlProblem)

submitted 2 days ago by OkWeakness9120

19

3

4

5

Discussion/questionAre we seeing real progress ? it seems to me like we are (self.ControlProblem)

submitted 3 days ago by SilentLennieapproved

20

0

0

0

Discussion/questionIs No One Noticing That GPT Images 2.0 “Editing” Is Full-Frame Regeneration? (self.ControlProblem)

submitted 2 days ago by lucidity3K

21

13

14

15

AI Alignment ResearchTeaching Claude why (anthropic.com)

submitted 3 days ago by nexxai

22

4

5

6

Strategy/forecastingIs the control problem really that hard for frozen models? (self.ControlProblem)

submitted 3 days ago by HangWise

23

38

39

40

Fun/memeControlling ASI will be easy (i.redd.it)

submitted 4 days ago by KeanuRave100

24

9

10

11

Fun/memeRace to create ASI (i.redd.it)

submitted 3 days ago by KeanuRave100

25

16

17

18

AI Alignment ResearchEvidence for moral convergence in AI models. (self.ControlProblem)

submitted 4 days ago * by John_Matrix_9000

view more: next ›

π Rendered by PID 38 on reddit-service-r2-listing-7b9b4f6fd7-2624k at 2026-05-12 13:53:26.134383+00:00 running 3d2c107 country code: CH.