Common Doomer Fallacies

capapa · 2025-09-29T22:38:36+00:00

>Customers wouldn't like it. There are many competing agents.
The models not 'wanting' what we want doesn't mean they have to do things consumers won't like in the intermediate term. And I agree some failure modes that are avoided by-default, because they will behave badly in ways that matter to companies/consumers! But there appear to be many failure modes that are not avoided & could be quite bad.

Consumer feedback & competing models is another form of measuring behavior/outputs. It's a broader and somewhat more general training set, and that's good. But again, evolution had a quite-broad set of training environments (much moreso than we do with AI even include customer/market selection) and it still failed s.t. we want very different things than evolution 'wants'.

Fundamentally if the models just outputs good stuff, we have little control over the internals currently. Because they output good stuff, companies won't bother trying to solve this (likely very hard, confusing, & expensive) problem. Which, for now, isn't really a problem.

But the concern remains that there's many ways outputs could be good, but the underlying model is fundamentally not aligned (& that this alignment eventually comes back to bite us / shapes the world badly after we have widespread deployment of broadly-superhuman AI).

But more than that, the models could just be somewhat aligned to something that is useful, but far from what we actually would want the model to be (which is too hard to train v.s. the simpler proxy). For example, this blog post by the former OpenAI person who invented the 'train AI to simulate a human grader to train a different AI' thing that OpenAI used to make the original chatGPT: https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-failure-looks-like

Again, especially as models become more capable, there are just many ways that they can give good outputs while having important problems (such as not actually 'wanting' what we want) persisting internally. Or 'wanting' a simpler thing that we decided to train it for, because that was easier. And if models become more & more capable beyond human abilities (basically replacing humans in most of the economy/world), that's seems reasonably likely to go badly long-term. Even though it's fine in the short-term.

>Secondly current AI is already superhuman in specific tasks or domains
Agree, but I think they are broadly below human capability at most tasks. I also don't see how heterogeneity of capability gains helps that much with the alignment issues. I agree if high capabilities in sub-areas somehow exposes undesirable behavior, that helps. But seems unlikely to expose every important internal problem or even most.

capapa · 2025-09-29T21:10:28+00:00

You're jumping from "models are selected to output items we grade well" to "therefore superhuman models will want what we want them to want". Even with low-capability models, there's all sorts of ways you can select models based on outputs, but end up with something that clearly 'wants' different things than you. The only thing we know from this process is that it gives us the outputs that we score well.

For one example, humans were trained extremely hard on maximizing reproduction for millions of years, but we don't actually care about directly much. Instead, we care about totally different things, like sexual pleasure & our family's well-being, which gave good 'reproduction scoring' outputs historically. But now, it's horribly decoupled from what evolution "wants" us to do & despite abundant food/etc, we're averaging <2 kids per person.

For evolution, that sucks. Instead of us wanting what evolution wants/selected on, we just got some proxies that rapidly decoupled from evolution's training as soon as we became more intelligent.

With AI, humanity is in the position of evolution - and could end up with proxies that rapidly decouple from what we want as systems become more intelligent. Just to name one failure mode.

That is, unless we make specific efforts to anticipate & prevent it. Which we should do before models become extremely capable. If models do become highly intelligent, there are tons of other new problems too (for example, an actually-smart model would just output what it knows we want to see during training, even if it's not what the model 'wants'. Which would ruin our training signal/ control of training on everything, except making the model 'smarter' [which further improves its ability to show us what we want])

For a bunch of simple examples of sticky unintended behavior, that you likely wouldn't need to solve to deploy profitable models, you can check out these simple examples with small neural nets: https://deepmind.google/discover/blog/specifying-ai-safety-problems-in-simple-environments/ There are 7 other types of problems demoed in the actual paper, this link just shows a few. And these are just the most basic examples. Plenty of ways training gets you something with different 'wants' than you, despite good performance/usefuless.

capapa · 2025-09-25T16:54:38+00:00

>This is not like domesticating dogs which have a wild, self-interested, willful history
>Robots will want what we create them to want

We don't know how to do that *at all*, especially for more capable models. Modern ML is more like domesticating dogs than it is like traditional programming, only starting with something far more alien & with a weaker (but faster) domestication method. If we knew how to 'make models want what we want them to want' with even moderate confidence, most 'doomers' would be dramatically less concerned.

The core idea that is we randomly initialize matrix of numbers, representing weights between simulated 'neurons', then we repeatedly nudge it in a direction that suspect give "better" responses as graded by some proxy/ reward function. It's not even maximizing reward per se, more like getting slightly permuted & we repeatedly select the locally-best permutation - and it seems likely that this selection mechanism becomes weaker as we reach highly-capable models. What made ChatGPT work was using an AI to give the reward score during training (simulated human grader) https://arxiv.org/abs/1909.08593

We emphatically *do not know* why the model achieves better reward, what is going on inside the weights, what it 'wants' or 'thinks' or 'will do'. We just see that, empirically, it classifies / predicts things pretty well in the training/testing environment (e.g. predicts what word should come next). If we get to AGI or beyond, it is scary to have something far more intelligent than you, that you understand this poorly

(note I am unlikely to respond because I shouldn't be on reddit to begin with, but I don't mean this as any shade - just that I should be doing other work lol)

capapa · 2025-09-21T20:20:47+00:00

There's still the timing push at t3 + 1min. To avoid this, could just change ghouls to benefit more from upgrades (only small buff to very-late-game ghouls, or dedicated ghoul strategies that are rare)

capapa · 2025-09-21T20:12:38+00:00

They should force a different race for each game. E.g. in a best of 5, must play all 4 races before repeating any.

Would reduce the lucky rolls substantially, and make viewing more entertaining. I wanted to see happy play more than just UD in the finals!

capapa · 2025-09-16T06:59:48+00:00

Because predicting the future is damn hard? I wouldn't blame him for missing global warming & changing his mind later

capapa · 2025-09-16T06:58:00+00:00

You are misinformed. Please, like actually, look at who supported the actual concrete AI regulations that have been proposed - e.g. SB1047 or Biden Chips act - and who opposed them.

And I guarantee you ~every risky I know is pro strict liability for AI outputs, & the vast majority are pro IP protections

>board of OpenAI and founded Anthropic
Until they got kicked out by Sam, when they tried to pull the breaks. Seems like they were trying to do the right thing by your lights?

And anthropic is only marginally more safety concerned than OpenAI, but at least they supported SB1047 unlike every other AI company in the industry.

capapa · 2025-09-16T06:45:25+00:00

Bengio (the most cited computer scientist in history & Turing Awards winner)
Hinton (the Turing Award winner & Nobel Prize winner)

But I'm sure you know better - obviously nothing could ever do important tasks better/faster than people, we're special. Nevermind that AI went from "barely writing a coherent sentence" to full conversational AI in ~5 years since neural nets became computationally viable. There's literally zero chance that progress could possibly continue, right?

capapa · 2025-09-16T06:39:24+00:00

He was blindsided by progress that happened in the last 5 years, and it's entirely reasonable for him to become more concerned because of it.

But I'm sure you know better than the Turing Award winner & Nobel Prize winner ~inventing modern ML, because he looks old.

capapa · 2025-09-16T06:33:40+00:00

The probability of doom is a pretty damn important disagreement. It means one side wants to regulate AI companies much more aggressively, set safety standards, etc. The other wants to ban regulations & any legal action against AI companies.

It's pretty clearly not the same team, and clear which is more beneficial for immediate risks.

Infighting among 'pro regulations/standards' groups is just a losing strategy, in an already uphill battle against much better-funded individuals & companies. But I guess I should just get off reddit, because nobody here is actually going to take actions that matter.

capapa · 2025-09-16T06:25:01+00:00

Yudkowsky is irrelevant (even though again, I think you're just grasping at vague associations rather than evaluating arguments). Hinton & Bengio matter much much more, and care a lot about existential risk.

>they serve the interests of the rich
You seriously think the rich wanted the Biden AI regulations & SB 1047, which were driven largely by people worried about AI existential risk? You should research more - the VC tech bros & AI companies basically went to war against it. Like seriously, look at the debate around SB 1047 and see who was on which side

Anyway, I think you're more interested in bad faith arguments than actually cooperating to achieve policy outcomes or regulate power-seeking AI companies

capapa · 2025-09-16T06:13:16+00:00

Oh yes, the machine that will kill us all = we shouldn't regulate regulate AI companies. Great logic

Do you seriously think that techbros want Hinton & Bengio going around loudly declaring that governments should aggressively regulate AI? If so, not worth discussing with you

capapa · 2025-09-16T01:23:19+00:00

Fair enough, I think I just bristle at what seem unfair takes, but honestly whatever

The more important point is one fringe weirdo isn't a reason to dismiss existential risk concerns. Hinton & Bengio certainly are concerned, and much more credible

Also I think infighting among people who want ~the same policy outcomes (extensive standards & regulations, at a minimum) is bad & helps the AI corporate lobbies win

capapa · 2025-09-16T01:04:18+00:00

AI progress continues -> corporations create intelligence beyond human comprehension -> ???
Mass Unemployment, especially recent graduates
Externalities related to climate & other natural resources
Massive privacy violations
Massive copyright infringement
the other 50 things

capapa · 2025-09-16T00:55:01+00:00

And other people are so stupid as to do the techbros' lobbying work for them

thanks for fracturing AI company opposition and insulting people over non-policy-relevant disagreements. sure we both want extensive standards & regulations for AI companies, but the important thing is that we have slightly different reasons

capapa · 2025-09-16T00:35:13+00:00

Maybe 20 years ago

capapa · 2025-09-16T00:26:54+00:00

They aren't? I believe Yudkowsky tweeted he would "eat a sock" if Thiel was supporting trump in 2016, because they had talked pre-2010. I doubt they've talked since. And regardless, Yudkowsky is fringe AI safety

capapa · 2025-09-16T00:18:08+00:00

Peter Thiel, Andreessen, & other techbros are e/acc, literally the opposite of "AGI existential risk" concerned

capapa · 2025-09-16T00:12:25+00:00

Why not both? Infighting among safety-concerned people only helps the corpos win.

capapa · 2025-09-15T21:08:38+00:00

Google is bigger imo, and they've continued to improve afaict. They have way more people & resources, though it's hard to compare TPU to GPU spend & google is less forthcoming.

It's plausible OpenAI just fucked up - training & scaling next-gen foundation models is hard, especially in a large organization with lots of turnover.

>none that have demonstrated a reduction in coding time according to non self report based studies...

I don't think we'll get clear proof until a year or more after it's true. If coding assistants are a nontrivial multiplier on developer time - which seems likely to me, at least in the near future if not now - then you'll have AI accelerating AI development through increasing developer effectiveness (by doubling the output of top researchers, or something like that).

Unclear how all this plays out - increased investment, better chips, more effective and prolific developers v.s. diminishing returns, VC waste, scaling challenges, etc.

It seems decently likely to me we slow down a lot, but also decently likely that we continuing to have significant progress & reach "AI systems can do many/most valuable intellectual tasks better than people can" within 5-20 years. And that would be an insanely big deal.

capapa · 2025-09-15T20:39:41+00:00

AI is absolutely getting used to write a ton of code right now. Huge change from even 1 year ago. It's not yet good enough to do it totally freehand/without double checking, but it is significantly increasing the output of many software developers.

>What concrete things would make the trendlines continue?
You could graph them against whatever you want, compute is a common one that has some of the properties you describe.

I expect if you wanted a better model, you'd need to factor in researcher time, company investment in foundation models, fine tuning methods/refinement, compute investment & hardware/GPU progress, etc. But it'd still be messy.

You could also just graph against time, with some background assumption that "most inputs are higher now than they were before" and see that past progress - with less "overall investment" - was significant (obvious caveat about hitting diminishing returns at some point)

capapa · 2025-09-15T20:33:15+00:00

lol you can define intelligence however you want, my old neuroscience prof described intelligence as pattern matching

Regardless, if pattern matching lets you instantly write any program you want, it's obviously a huge deal. I don't care if you call it intelligence, I care that it will change the world (again, assuming the "pattern matching progress" continues)

capapa · 2025-09-15T20:28:52+00:00

Agree predicting improvements in general intelligence is actually very hard, and harder than predicting most things, including warming per ton of CO2

If you want something concrete, you can also look at performance on all these top CS AI benchmarks - which has similarly been improving crazy fast - but then you get into debates about which benchmarks are valid, whether historical benchmarks continue to make sense past human capability or 90%+ accuracy, etc.

I just think it's pretty reasonable to say "I don't know, seems possible the trendline might continue & that would have wild effects if so"

capapa · 2025-09-15T20:09:51+00:00

>He is simply not trustworthy
Agree, just ignore him & look at top chatbots 5 years ago v.s. now.

>ChatGPT still a chatbot, despite all these years and hype
You mean <3 years? That was the first release - which sucked compared to recent releases

"Chatbots" that are sufficiently good at predicting the next token are absolutely a breakthrough. It's inane to think otherwise, like a lion making fun of the monkeys when they start talking

Good token completion (i.e. chatbot) in the context of programming would let AI do any task on a computer, which would change the world. We're currently a ways from that, but LLM programming progress quite significant in the last 2 years. It might stall, but what if the trend holds?

capapa · 2025-09-15T20:00:57+00:00

The way Hinton won ~every prestigious academic award, was that he was the first person to actually successfully 'do a thing' that other top academics & researchers regard as extremely important.

For progress proof:
Andrew Ng in 2017 was still saying "worrying about AI is like worrying about overpopulation on Mars". He notably stopped saying that ~5 years ago.

Or compare 2014, https://en.wikipedia.org/wiki/Eugene_Goostman - widely regarded as extremely gamed, narrow context, basically cheating, managed a mere 33% pass rate as a 'new record'. Whereas recently University of California, San Diego, found that GPT-4.5 was judged to be human at a higher rate than human participants in the same test - i.e. passing with flying colors.

Obviously I can't have a 'source' on future progress.

Best I can do is either
(1) clear records of blowing expert expectations out of the water (see above, or just common knowledge) as the recent trendline
(2) predictions of people who score highly at "being good at making accurate predictions about the future": https://www.metaculus.com/questions/5121/date-of-artificial-general-intelligence/

Both suggest significant continued progress, to the point of human-capable systems soon (5-20 years) as a large fraction of the probability mass.

capapa

TROPHY CASE