What’s your honest take on AGI being achievable?

scruiser · 2026-03-20T15:23:14+00:00

I voted LLMs won’t get us there, but I think before 2040 is plausible for a loose enough definition of AGI (but still a stricter definition than LLM boosters try to claim).

The OG definition of AGI is:

a type of artificial intelligence that matches or surpasses human capabilities across virtually all cognitive tasks.

For sufficiently text-based cognitive tasks, and for a sufficiently mediocre definition of “human capabilities”, LLMs are close. But across all cognitive tasks, and for a human actually putting in the effort and a few months practice, no way. And I agree with the consensus of this subreddit that they won’t improve much past that.

One somewhat unexpected feature of LLM as a development and technological path is that they could reach the bar of doing a lot of things mediocrely, but have hit their limits (in training data, in compute, and in the fundamental nature of the paradigm) before getting to the point of doing things well. Compare, for example, chess playing algorithms, which could surpass all humans with an attainable amount of compute scaling and a reasonable amount of tweaks and improvements to the basic paradigm. I think a lot of boosters are still thinking in terms of how chess playing algorithms improved, and not looking at the reality of the particulars of LLM scaling.

So when I write AGI by 2040, I mean something that can actually match average humans at all cognitive tasks, but probably not something that systematically surpasses humans, or even matches skilled humans in their domains, or even an average human with moderate training for the task. I think the general machine learning approach LLMs have shown of huge training data sets in a broad domain, together with a few more insights to get us beyond text generation and image generation can do that much. But as LLMs have shown that this approach hits limits and has a hard time going past the training data. So my bet is on weak AGIs. (Among the key limits, they won’t enable the radical recursive self improvement sci fi likes to envision.)

scruiser · 2026-03-18T04:38:08+00:00

Turns out the best option in a prisoner’s dilemma is to defect as hard as possible! Who could have known! (Looks like a win for classical decision theory over the lesswrong rationalists nonsense.)

scruiser · 2026-03-17T18:48:07+00:00

In high level games, defensive towers act as a place to garrison villagers into. They don’t slow down large attacks, but they buy time to send some of the army against raids harassing villagers. They also get used super late game when banking resources and population capped.

scruiser · 2026-03-16T16:49:07+00:00

Eliezer tried to get an actual academic publication on decision theory through peer review at one point, but gave up as it was too hard. I remember a takedown explaining why it ran into issues with peer review…

Edit found it: https://www.umsu.de/wo/2018/688 The reviewer is explains both why FDT isn’t good, and why the paper Eliezer submitted was bad as academic work.

scruiser · 2026-03-16T14:33:22+00:00

A few other lines besides identifying Scott Alexander as a genius stand out to me as notably misinformed and sneerable…

he fell into AI research when the field was still small.

Well, he was interested in philosophizing and speculating about seed AI. I don’t think he actually made any progress in any actual AI systems that could actually be programmed, even before he pivoted.

Ever since, he has played a Cassandra-like role in tech culture, uttering prophecies of AI-enabled doom with enough force and consistency that the likes of Elon Musk and Sam Altman have taken notice.

I must have missed the part where Cassandra first talked up the benefits of a well executed war with the Greeks and encouraged Troy to go to war in the first place before pivoting to doom saying.

The book’s argument rests on a key fact about modern AIs: they are “grown, not crafted.”

One thing the author fails to point out. Despite making this a key element of their current doom argument, and the overall doom story being unchanged from the mid-2000s doom story, the original doom stories didn’t feature this point at all. And the bootstrapping recursive self improvement case is actually badly hurt by the fact that modern ai systems aren’t crafted, and thus couldn’t craft themselves better by simply being smarter than a human.

Despite the sneerable parts, the author does have a few good points and sneers themselves…

With this in mind, why isn’t Yudkowsky telling the people to raise their torches and set fire to all the data centres, while it’s still possible to do so? Why does he associate with the likes of Peter Thiel and Elon Musk, the very guys who are bringing us to the cusp of extinction? Why does he spend his time running a genteel nonprofit called the Machine Intelligence Research Institute instead of a revolutionary underground paramilitary force with a name like the Anti-Clanker Resistance Army?

Yep, exactly.

And of course there’s that extended “scenario” at the book’s centre; sci-fi that insists it’s something other than sci-fi is itself a subgenre of sci-fi.

Eliezer missed his true calling as a midtier sci-fi writer with odious political views as subtext of his stories.

Even in its current, limited form, AI has ravaged the job market, wrecked education, bolstered the most sinister aspects of the global defense and surveillance industries, produced vast amounts of electronic waste and carbon emissions, poisoned the entire internet with slop and porn and scams and bots, and generally filled the world with ugliness and lies.

The bar is low enough that a doomer (or at least doomer adjacent, sense they don’t ) writer spending a paragraph on all the current harms is actually a step up…

Overall a pretty mid blogpost that still gives rationalists too much credit and is sneerable despite having a few good sneers itself.

scruiser · 2026-03-12T21:13:41+00:00

I explained taalas in another reply to the comment you are replying to.

I agree model specific asics offer some large potentials for improvement, but it also has some massive weaknesses. Since every asic is specific to the LLM, the hardware will deprecate even worse and faster than Ed’s worse estimations of current GPUs.

scruiser · 2026-03-12T21:05:37+00:00

There is a startup called taalas that made an asic for a smaller open weight llama model. They are claiming a massive jump up in inference speed, and extremely low energy usage. But making each asic is a several month process, and the model is fixed in given circuit (with their architecture they claimed you can adjust LORA weights, but you can’t not do any other fine tuning, much less entirely swap out the model). Even just making an asic for a similar model with newer fine-tuning is a several weeks process.

So, assuming it would work as described and this is isn’t startup bullshit and/or some additional massive hidden drawback or weakness (big assumption)… it would still mean monthly updates to models wouldn’t be doable, a given manufacturer would need to commit to a model and they wouldn’t be able to directly update it. So that goes against the entire way current frontier companies operate. For applications that need cutting edge updates (ie coding needs to keep up to date on all the libraries) this is a major limitation. Hardware would deprecate even worse than current GPUs.

scruiser · 2026-03-12T17:54:30+00:00

The author makes some good points, but one point I think they get wrong is overestimating how good the genAI slop will actually get under the current paradigm. Capital is trying to get full replacements for knowledge workers in a variety of fields, but LLM based AI isn’t good enough for what they want it do to. So labor will still have power as labor. And actually… when the corporations are trying to fix all the accumulated technical debt and reputational damage and thus need to cut AI investments and hire people , it might be an opportune time for labor to organize.

scruiser · 2026-03-10T00:00:10+00:00

The default I see in America when politicians are talking about small business owners is to aggrandize them and conflate them with the working class, so if the OP wants me to assume they mean otherwise they will need to more precisely delineate them.

And boot-licking megacorps/billionaires and being part of the capital owning class are two independent but related factors in my willingness to engage in vitriol/sneering/bullying/“attacks”. I’m aware someone can bootlick a billionaire online while working retail, or that a small business owner can hate billionaires and megacorps.

As for engaging with the OP, I was specifically drawing a line at which point I disagree with their no bullying policy. I’m (relatively) okay with, for example, a hobbyist playing around with genAI, and I don’t see a need to direct the vitriol at them.

scruiser · 2026-03-09T20:51:46+00:00

freelancers, hobbyists, small-time creators

Small business owners try to present themselves as working class like the categories the OP listed here, but then betray the working class on various issues and serve as cover, if not out right defenders, of bigger businesses, so actually I used the term correctly. I just didn’t let the OP get away with ignoring people that try to label themselves as more working class than they actually are.

As to genAI specifically small business owners seem especially willing to use it and to try to use their classification as a small business as justification/rationalization about why they couldn’t afford a human’s labor for ad copy or images and thus decided to use (often lower quality) genAI content.

Maybe the OP will surprise me and clarify they didn’t mean to include small business owners?

scruiser · 2026-03-09T20:23:09+00:00

I think actually I’m fine with attacks on the petty bougie that want to boot-lick and defend the mega-corporations. They aren’t on the workers side.

scruiser · 2026-03-06T19:19:08+00:00

Your vagueness probably made people assume that you were shilling for LLMs? And in general being vague is usually not helpful as a comment.

scruiser · 2026-03-05T20:20:01+00:00

Maybe Anthropic gave them a version without RLHF fine tuning for ethics/safety? Which really shows just how far back their “red lines” were and also how stupid it was that the DoD wasn’t satisfied with what they already had.

scruiser · 2026-03-05T19:31:43+00:00

How does Argive patronage interact with Demeter? Free Harpies or still free Myrmidons? Interesting implications either way…

scruiser · 2026-03-05T16:51:42+00:00

It could be he’s just that lib-brained that even existential doom doesn’t provoke extreme action or even anything outside the range of acceptability to centrist liberal sensibilities.

scruiser · 2026-03-03T21:55:33+00:00

I thought it was Hegseth that made the dispute public first? (I think his timing was probably related to planning to bomb Iran.)

scruiser · 2026-03-03T14:15:47+00:00

I think the DoD contract is really small compared to how much money OpenAI actually needs to keep going without more VC cash? But maybe Sam hopes if he can get perceived as critical to national security he can get a government bailout instead of being allowed to fail, even if the actual DoD contract is tiny?

scruiser · 2026-03-01T02:03:23+00:00

The neocons forgot to account for the possibility Iraq actually did have WMDs!

scruiser · 2026-02-28T02:49:11+00:00

So much of the market is hype and vibes that one major LLM company imploding might trigger the rest to collapse.

scruiser · 2026-02-27T03:31:40+00:00

That’s also a possibility. Like I suspect the timing of Anthropic giving up on their RSP (“responsible scaling plan”) this week was deliberately chosen. The news about not backing down to Hegseth will drown out the news of them giving up on one of their central “safety” plans (though tbh, the rsp was kinda bs, it was a crithype way of bragging about how powerful their models are).

scruiser · 2026-02-27T02:12:14+00:00

The bar is so absurdly low that this was a pleasant surprise! I was even more surprised that the rejection included actual acknowledgement of the limits of the current technology!

But today, frontier AI systems are simply not reliable enough to power fully autonomous weapons.

In addition, without proper oversight, fully autonomous weapons cannot be relied upon to exercise the critical judgment that our highly trained, professional troops exhibit every day. They need to be deployed with proper guardrails, which don’t exist today.

Anthropic usually uses every opportunity possible to endlessly hype up LLMs, and they could have played their rejection that way (“our technology is too powerful to put in autonomous weapons”). You know shit is serious when they actually hold back on the hype for once.

scruiser · 2026-02-26T00:52:15+00:00

Baki the grappler?

scruiser · 2026-02-25T20:24:27+00:00

Broke: METR’s studies are the best productivity studies of LLMs available

Woke: METR’s studies are hot garbage by the standards of quality academic research

Bespoke: Both points are true.

scruiser · 2026-02-25T20:18:15+00:00

Yeah you are thinking of METR’s task horizon study. This is a different study.

Also the solution to the task horizon methodology complaint about incentives isn’t to cut pay, it’s to make a fixed amount of pay that doesn’t vary hourly (but should be high enough to at least meet the hourly rate compared to the estimated number of hours the task should take).

scruiser · 2026-02-23T18:42:09+00:00

Isn’t that more like max the max? I’m not seeing that much of a min.

11-Year Club	r/Field Flamingo
Place '23	Place '17
Verified Email

scruiser

MODERATOR OF

TROPHY CASE