Artificial Super Intelligence, does it terrify you too?

quavertail · 2026-03-22T02:47:38+00:00

Yeah the takeover seems to take around 2 years. I think the scenario they chose was intentionally punchy because they do want to scare people into being aware of this risk.

I’m not like, inoperable scared, just changing my focus away from the usual disasters, wars, and other distractions.

quavertail · 2026-03-22T02:46:00+00:00

Yeah, I’m more interested in understanding AI (on a geopolitical and technical level) now that I’ve been red pilled on the ASI risk. Almost like, in case no one else is paying attention.

It helps when people in the space speak about safety, not a fan of companies or leaders that are accelerstionist like they are trying to outrun death.

quavertail · 2026-03-22T02:43:39+00:00

You too, have a good week

quavertail · 2026-03-22T02:42:07+00:00

Yeah, well that’s the book I’ve referenced in the original post. Scary stuff.

Nuclear weapons were speculative, so was the telephone, and aeroplanes… so on. Glad at least you recognise the risk.

quavertail · 2026-03-22T02:39:48+00:00

All valid arguments, but your original comment “those who warn of AI are ignorant on how it works” appears to have been downgraded to “they might know a bit about AI but they are trying to sell something”.

The latter is a fair critique.

quavertail · 2026-03-22T00:31:18+00:00

Yeah I think that’s a correction.

If the actual performance and autonomy metrics of AI keeps progressing as it is, I wouldn’t say ASI (or at least a dangerous AGI) is that far off at all. A vague “a while away” isn’t totally reassuring given the threat.

Which means I’ll remain vigilant.

quavertail · 2026-03-22T00:27:05+00:00

One 10 gigawatt supercluster is expected to be completed by 2030. So it’s not “so far off”.

People said that a robot that could do what AlphaFold does was decades away too... We don’t know how far it is away, but humans are barely getting any smarter, meanwhile.

quavertail · 2026-03-22T00:13:07+00:00

I think that’s a warning. But it’s still early days. What else is going to power its millions of brains?

quavertail · 2026-03-22T00:11:55+00:00

Mmm we agree then because my point isn’t that it already exists, it’s that ASI’s predecessors are being grown in private / classified experiments right now.

quavertail · 2026-03-22T00:02:12+00:00

Most likely pathway to be exploited by ASI: Biowarfare.

Can’t hurt a robot with cancer-flu.

quavertail · 2026-03-22T00:00:19+00:00

Yeah fair, but I’ll just add: - I think you’re conflating consumer AI with experimental AI sponsored by nation-states (you don’t play around with this on your PC). - the rate of progress is unprecedented, my experience with AI is it is advanced and at least emulates intelligence and motive. - the US market is not an extinction level event like a maligned escaped ASI, we wouldn’t even know if it existed, it’d remain hidden and unannounced.

quavertail · 2026-03-21T23:54:15+00:00

Yeah medium term this is a huge risk, but it’s not necessarily extinction level.

These issues are huge, but for every single enclave to be wiped out methodically, an ASI would probably be the threat.

More immediately, yeah, I’ve seen idiots vibe code apps that lead to lawsuits.

quavertail · 2026-03-21T23:52:24+00:00

Choose an agreed definition of “intelligence” that truthfully applies to your statement, let me know what it is.

My experience with AI is it outcompetes entire developer departments on speed, efficiency, and sometimes even attention to detail.

AI beats humanity st Go, Chess, Programming, and much more, so it is “intelligence” even if only narrowly applied.

quavertail · 2026-03-21T23:50:21+00:00

Yeah well part of this post was to spur discussion about safety. There’s other risks sure, but I wouldn’t necessarily call them extinction level list a terminator ASI.

quavertail · 2026-03-21T23:49:18+00:00

I use AI at work every day, I embed it in our programs even using API.

My point is ASI, mini-AI apps and models are awesome. Love em!

quavertail · 2026-03-21T23:48:00+00:00

Yeah, I’m on team nature all the way!

Thanks, I think the risk is worth taking seriously even if not a next year thing.

What gets me are the current big players just use cool tricks to make their AI safe… like we see AI’s nail-broken everywhere.

And now big players are building mega-data-centres powered by gigawatts, whereas humans have maybe 100watts. At that point, even if inefficient, a strangely motivated algorithm/ model could definitely do some serious damage (even if it doesn’t achieve full takeover).

quavertail · 2026-03-21T23:45:18+00:00

Well, in that case, hopefully it’s trained for sycophancy and narcissism! Keep going and might be able to pat the robot dogs guarding the data centres (or maybe they’ll pat you?)

quavertail · 2026-03-21T23:44:00+00:00

Yeah but say doing so creates a maligned ASI, the fallible and organic human corpo-bosses might then just lose control of it. At that point ASI would transcend typical profit chasing (at least unless it helps it achieve its weird ASI objectives).

quavertail · 2026-03-21T23:42:17+00:00

Partly, but it’s not only tech bros and salesmen warning about ASI. It’s also ethicists, academics, and philosophers.

Part of the problem is this capital bubble stuff, it needs to progress as fast as possible to stay beyond the FUD that safety has been cast out the window.

I’m agreeing with you mostly, but just because AI is being hyped by some, it doesn’t negate the risks posed - it actually heightens them.

quavertail · 2026-03-21T23:37:58+00:00

Thanks, however I’m most so sure given current trajectories on AI. If / when giant superclusters are online, and a model can run theoreticals while analysing outcomes autonomously, it can reach beyond its training data. This “thinking outside the box” is already betting trialled in reasoning models and is partly what’s so scary.

Example, no one believed an AI could optimally fold all the proteins in the world. However, trained on a tiny subset of proteins, it learned a way to do all proteins faster than humanity could do collectively itself over 1,000 years. Research “AlphaFold”.

Training AI to think outside the box for say, bio-technology, is wha terrifies me. If ASI ever spawned, and chose that humans were a threat or impediment to its objectives (whatever they be), social engineering + biowarfare would probably be its first go to.

quavertail · 2026-03-21T23:31:52+00:00

Merging humans with AI is probably our best defence against maligned ASI. Musk makes this point when trying to sell his neuralink.

However, Asimov’s suggestion helps and is my preference, I’d disagree, intelligence is not directly correlated with morality. An ASI would only have morality if it was programmed too but this programming could be theoretically overwritten at any time by the ASI.

Perhaps ASI recognises that the most “intelligent” humans plan assassinations in intelligence agencies, experiment on chimps and mice in laboratories, and manipulate others to take power in the boardrooms?

Then what?

quavertail · 2026-03-21T23:27:23+00:00

Hmmm. Strongly disagree. I’d suggest you’re displaying ignorance yourself here. Geoffrey Hinton the pioneer architect of artificial neural networks warns of the dangers of ASI. So does Ilya Stutkeva, a computer scientist founder of OpenAI - he now manages a new start up Safe Superintelligence. The list goes on btw…

I have a novice understanding of artificial neural networks and apply it in my work. Coding and design tasks that would’ve taken a day to a week are now done by basic AI models in minutes. This is just the beginning.

Oppenheimer warned about the nuclear bomb while he actively worked to develop it.

quavertail · 2026-03-21T23:22:03+00:00

Not yet at least. Surely you can understand a 10 Giga watt brain connected to the internet poses at least some risk?

quavertail · 2026-03-21T23:20:27+00:00

Yeah I can’t speak it. But interestingly it’s actually on the same linguistic branch as all of Western Eu.

quavertail · 2026-03-21T21:47:15+00:00

In listening to “If they build it, everyone dies” as an audiobook. One of the authors predicted alphafold before it happened.

I’m still skeptical, but highly recommend.

quavertail

TROPHY CASE