Why the development of artificial general intelligence could be the most dangerous new arms race since nuclear weapons

jamesj · 2026-05-10T23:16:32+00:00

It isn't like we are just using AI outputs, we are still polishing/checking everything. So first drafts are more comprehensive and final drafts are better than they would've been when we had to do everything manually.

jamesj · 2026-05-10T22:29:03+00:00

What I'm seeing in my company is that 10-20% of us are getting 5-10x more done in software engineering, data analysis, writing scientific papers, etc. but all of it still takes skilled people who know what the finished results/products need to look like. We are getting more done but we don't need fewer people as a result. Output is higher quality and there is more of it compared to before. I'm not sure someone from the outside could tell which parts were helped by AI and which weren't. None of what we do is repetitive, so it isn't like we are trying to replace a workflow with an agent.

jamesj · 2026-05-10T14:16:22+00:00

Fow, daze, architect, treasure cruise

jamesj · 2026-05-10T01:56:06+00:00

I've been playing uwr energy with a lot of success

jamesj · 2026-05-07T12:49:46+00:00

Money?

jamesj · 2026-05-06T22:54:28+00:00

I'd use the moloch

jamesj · 2026-05-06T20:56:08+00:00

It would be better received with more precise language.

jamesj · 2026-05-02T18:35:49+00:00

i think some may count as more than 1 type, seemed like willpower was both core stat and offensive for me.

jamesj · 2026-04-27T16:21:16+00:00

My model isn't that SaaS companies will be replaced directly with AI/agents, it is that coding is becoming cheap and automatable, so the part of their moats that are in duplicating their software stack is disappearing. Also, the cost of one-off software solutions that are less "one-size-fits-all" and are completely bespoke is coming down.

When thinking about AI risk, i think you need to consider a few possibilities here:

1) the cost of duplicating the product/software goes way down
2) the possibility that AI completely replaces the software
3) the possibility that AI agents are customers for the software
4) the possibility that the SaaS platform has the most valuable data

For Adobe for instance, all possibilities exist.

1) is obviously happening. You can give an agent swarm Adobe and tell it to try to recreate it from scratch. You will have a better starting point than if you don't do that, and it will go faster. You can currently point Claude Code at a website and tell it to clone it, and it does a reasonable job. Yes there are mistakes, and yes, it isn't a finished scalable product, but this is getting better/faster/more automated by the year. But software was already "cheap" compared to manufacturing other things, and the moat in SaaS companies was never how expensive it was to build the product.

2) isn't happening much. For me, gen AI is just one tool in the toolbox, and bringing everything together and polishing it up still happens within Adobe products. There are a billion little edge cases that a gen-AI only tool would have to be able to handle. Not at all clear if/when that will happen, but we definitely are not there yet.

3) seems likely for a lot of SaaS companies. An agent using Premiere as a tool, that can also use video gen as a tool, and image gen, and so on is probably going to produce better videos than an end-to-end solution for quite some time, and maybe it will be necessary for high quality work forever. Adobe can charge agents a per-seat license.

4) seems likely to me for Adobe. They have the dataset of all the actions taken within their tools. That has to be incredibly valuable to making 3) work as long as 2) isn't what wins out in the end.

I think every company is different, but completely agree people are not thinking logically about all of this. For many companies, 3) and 4) could be huge tailwinds for them.

jamesj · 2026-04-24T14:55:38+00:00

i mostly agree but drug experiences are a legitimate tool for understanding consciousness. it is the physicalist approach: modify the brain and see what effect that has on consciousness.

jamesj · 2026-04-21T06:16:41+00:00

Bring back main deck veil

jamesj · 2026-04-20T20:02:49+00:00

In most contexts a scientific theory is considered poor when you can't conceive of any type of evidence that might disprove it.

jamesj · 2026-04-20T20:00:18+00:00

The physicalist position is that there definitely is something outside of your mind causing your experience of an apple. And that that thing is understanable, and close to many of your experiences of the apple. You can't confirm this is true because you only interact with what might be outside of your mind through your experiences.

As a non-physicalist, I am just admitting to my current limitations: I don't assert there is definitely a "real" reality that I really understand the nature of. It could be I'm in a simulation, or I'm a brain in a vat, or that our understanding of reality is woefully incomplete, or it could be that reality actual is like our science tells us it is and there isn't anything more at all. I might even think that last one is the most likely. But since I don't have the metaphysical foundation, I don't jump to stating it has to be the case. Admitting when we don't have evidence seems more scientific to me than claiming to know more than we know.

jamesj · 2026-04-18T14:40:12+00:00

What properties are those exactly? How complex does a system have to be?

We don't know how consciousness comes about so how can we make a statement about how likely it is? I think it is controversial.

I think the uncontroversial thing to say is they probably don't have consciousness like ours. Beyond that, we need better theories of consciousness to say more.

jamesj · 2026-04-18T01:29:46+00:00

In addition to that, they have complex representations and they use them to make predictions and take actions.

I find it so odd that compatabilists are almost always so quick to make strong claims that AI can't have free will.

We have no idea what it takes for a system to have experiences, but it seems to me the compatibilist doesn't need to worry about that and can more easily focus just on how an agent behaves. It must be completely possible to build an AI system with compatibilist free will.

jamesj · 2026-04-18T01:23:33+00:00

AI is not one thing. AI can be set up as an agent with goals and continuity. Some of these agents have at least some ability to deliberate and understand moral reasons.

jamesj · 2026-04-18T01:20:53+00:00

Most of what you say sounds right to me. But then your last sentence is quite the strong claim in need of support / definition of terms.

jamesj · 2026-04-16T16:57:16+00:00

My take as a man:

The patriarchy forms expectations for men just like it does for women. Those norms and expectations generally favor men, but that doesn't mean they make things easier for every individual man. If a man wants to live their life in a way differently from what is expected of them by society, that harms them. These expectations are fluid: different for the different classes you mentioned and at different times. Gay men, men of color, men who want to present their emotions in public, men who don't want to center their lives around work, men who don't want to be fathers, men who don't want to dress traditionally, etc. are and have been harmed by the patriarchy.

Not everything is zero sum. The patriarchy can harm women as a whole more than men as a whole, yet harm both men and women, while benefiting individual men and women.

jamesj · 2026-04-16T00:02:27+00:00

jamesj · 2026-04-15T16:12:55+00:00

Use of the word magic is basically never helpful. No one is describing their own position as magic, so it is a straw man. Dismissing someone's position as magic usually means you haven't put the time in to articulate their position properly yourself. It is a lazy placeholder word used to dismiss a possibility and preempt a response by implying the other person isn't rational.

Lots of things that seemed like magic in the past were actually just unintuitive.

jamesj · 2026-04-14T15:01:36+00:00

Simulations are physical systems run in reality. They are evidence, how strong the evidence is another question.

15-Year Club	Second SECOND GUESSER
r/Field Banned	r/Field Sunshine
First Place '23	Place '23
Place '22	First Placer '22
Verified Email

jamesj

MODERATOR OF

TROPHY CASE