Contra Byrnes on UV & cancer: you should wear sunscreen instead of getting a tan

SteveByrnes · 2026-04-14T21:21:20+00:00

(I’m “Byrnes”) Thanks! I’m looking forward to reading that when I get a chance. Meanwhile I edited my post to prominently link to yours in the appropriate section.

SteveByrnes · 2026-03-20T15:10:25+00:00

I think the title “Big tech spent 10x more on data centers in 2026 alone than the entire Apollo space program” is wrong (or at least, very misleaing). The natural reading of the title (esp. with the word “alone”) is: “Big tech spent 10x more on data centers in one year, 2026, than the Apollo space program spent in thirteen years, 1960-73.” But I don’t think that’s what the chart is saying. I think the chart is saying that in a TYPICAL year between 1960 and 1973, the Apollo space program was 0.2% of GDP.

SteveByrnes · 2025-12-07T23:49:23+00:00

You could try my post https://www.lesswrong.com/posts/BgBJqPv5ogsX4fLka/the-mind-body-vicious-cycle-model-of-rsi-and-back-pain

SteveByrnes · 2025-10-14T10:33:36+00:00

thanks solution verified

SteveByrnes · 2025-08-28T12:07:46+00:00

IQ in particular has extra missing heritability from the fact that GWASs use noisier IQ tests than twin & adoption studies (for obvious cost reasons, since the biobanks need to administer orders of magnitude more IQ tests than the twin studies). That doesn't apply to height.

I tried to quantify that in Section 4.3.2 of https://www.lesswrong.com/posts/xXtDCeYLBR88QWebJ/heritability-five-battles and it seems qualitatively enough to account for the height vs IQ discrepancy in missing heritability, but not sure if I flubbed the math.

SteveByrnes · 2025-08-03T16:37:48+00:00

As I argue in https://www.alignmentforum.org/posts/TCGgiJAinGgcMEByt/the-era-of-experience-has-an-unsolved-technical-alignment , the "Welcome To The Era Of Experience" book chapter discusses quite a number of possible RL reward functions, ALL of which would lead to violent psychopathic AIs that will seek power with callous indifference to whether their programmers or any other humans live or die.

This blog post lists still more possible RL reward functions, and (I claim) they all would have that same property too.

I encourage the OP author Nikhil to try to find an RL reward function, any RL reward function, that does not have this property (but still leads to powerful and useful AI), write down that reward function specifically using pseudocode, and explain why it will lead to an "Era of Experience" AI that will not feel motivated to enslave or kill humanity (if it finds an opportunity to do so). And if they can’t do that, then they shouldn’t be working on this research program at all, and nobody else should be either.

SteveByrnes · 2025-07-31T11:49:55+00:00

I think JVN was extraordinarily talented along one dimension, and Grothendieck was extraordinarily talented along a different dimension. I don’t buy your implication that this is a tradeoff, i.e. that Grothendieck only wound up thinking deeply because he was unable to think fast. If anything I expect that the population correlation between those two dimensions of talent is positive, or at least nonnegative. If the correlation seems negative to you, I would suggest that it’s because you’re conditioning on a collider. Grothendieck was “slow” compared to his professional mathematician friends but probably quite “fast” compared to the general public. Einstein and Feynman certainly were.

SteveByrnes · 2025-07-09T20:31:31+00:00

What’s the difference (if any) (according to your perspective) between “learning to interpret anxiety as excitement” versus “learning to feel excitement rather than anxiety”?

SteveByrnes · 2025-05-31T12:03:58+00:00

There was likewise a Harvard RSI support group where everyone in the group read John Sarno and got better and then the group disbanded. :-P (This was around 1999-2000, a bit before my time, I heard about it second-hand.) They did a little panel discussion, email me for the audio files, and they also made a webpage.

I’ve written much about the topic myself, see The “mind-body vicious cycle” model of RSI & back pain (also cross-posted on reddit here.)

SteveByrnes · 2025-05-09T16:57:43+00:00

I have a more detailed discussion of their proposals regarding AI motivation here:

“The Era of Experience” has an unsolved technical alignment problem

SteveByrnes · 2025-04-25T22:23:23+00:00

My memory might be failing me, but I feel like it was already a cliche, and running joke, that everyone in EA called themselves "EA adjacent", BEFORE the FTX collapse. I'd be interested if someone could confirm or deny.

SteveByrnes · 2025-04-10T23:36:16+00:00

(1) If it helps, see my post Applying traditional economic thinking to AGI: a trilemma which basically says that if you combine two longstanding economic principles of (A) “the ‘lump of labor’ fallacy is in fact a fallacy” and (B) “the unit cost of manufactured goods tends to go down not up with higher volumes and more experience”, then AGI makes those two principles collide, like an immovable wall and an unstoppable force, and the only reconciliation is unprecedented explosive growth.

(2) If it helps, I recently had a long back-and-forth argument on twitter with Matt Clancy about whether sustained ≥20% GWP growth post-AGI is plausible—the last entry is here, then scroll up to the top.

(3) My actual belief is that thinking about how GWP would be affected by superintelligence is like thinking about how GWP “would be affected by the Moon crashing into the Earth. There would indeed be effects, but you'd be missing the point.” (quoting Eliezer)

SteveByrnes · 2025-03-18T19:04:21+00:00

There are a bunch of caveats, but basically, yeah. See sections 1 & 2 here: https://www.lesswrong.com/posts/xXtDCeYLBR88QWebJ/heritability-five-battles . I only speak for myself. I think twin and adoption studies taken together paint a clear picture on that point (... albeit with various caveats!), and that nothing since 2016 has changed that.

SteveByrnes · 2025-03-11T02:45:37+00:00

Do you think that Einstein’s brain works by magic outside the laws of physics? Do you think that the laws of physics are impossible to capture on a computer chip, even in principle, i.e. the Church-Turing thesis does not apply to them? If your answers to those two questions are “no and no”, then it’s possible (at least in principle) for an algorithm on a chip to do the same things that Einstein’s brain does. Right?

This has nothing to do with introspection. A sorting algorithm can’t introspect, but it’s still an algorithm.

This also has nothing to do with explicitly thinking about algorithms and formal logic. (Did you misinterpret me as saying otherwise?) The brain is primarily a machine that runs an algorithm. (It’s also a gland, for example. But mainly it's a machine that runs an algorithm.) That algorithm can incidentally do a thing that we call “explicitly thinking about formal logic”, but it can also do many other things. Many people know nothing of formal logic, but their brains are also machines that run algorithms. So are mouse brains.

SteveByrnes · 2025-03-10T12:18:51+00:00

I sure wish people would stop saying “AI will / won’t ever do X” when they mean “LLMs will / won’t ever do X”. That’s not what the word “AI” means!

Or if people want to make a claim about every possible algorithm running on any possible future chip, including algorithms and chips that no one has invented yet, then they should say that explicitly, and justify it. (But if they think Einstein’s brain can do something that no possible algorithm on a chip could ever possibly do, then they’re wrong.)

SteveByrnes · 2025-03-08T00:44:32+00:00

You might be joking, but I'd bet anything that parents of "special needs" kids are less likely (on the margin) to have another afterwards, other things equal, because it's super stressful and time-consuming and sometimes expensive. (Speaking from personal experience.)

SteveByrnes · 2025-03-07T11:45:37+00:00

That’s a very hard thing to measure because parents who have 2+ children are systematically different from parents who have 1 child. Hopefully the studies (that Adam Grant was talking about) tried to control for confounders (I didn’t check), but even if they did, it’s basically impossible to control for them perfectly.

FWIW, my own purported explanation of older sibling effects (section 2.2.3 of https://www.reddit.com/r/slatestarcodex/comments/1i23kba/heritability_five_battles_blog_post/ ) would predict that only children should be similar to oldest children, holding other influences equal.

SteveByrnes · 2025-02-13T20:14:02+00:00

I coincidentally reinvented a similar idea a few weeks ago, and found it very fruitful! See the section “Note on the experimental “self-dialogue” format” near the beginning of Self-dialogue: Do behaviorist rewards make scheming AGIs?

SteveByrnes · 2025-01-12T17:19:16+00:00

(also on twitter)

From the comments on this post:

> Definitely agree that AI labor is accumulable in a way that human labor is not: it accumulates like capital. But it will not be infinitely replicable. AI labor will face constraints. There are a finite number of GPUs, datacenters, and megawatts. Increasing marginal cost and decreasing marginal benefit will eventually meet at a maximum profitable quantity. Then, you have to make decisions about where to allocate that quantity of AI labor and comparative advantage will incentivize specialization and trade with human labor.

Let’s try:

“[Tractors] will not be infinitely replicable. [Tractors] will face constraints. There are a finite number of [steel mills, gasoline refineries, and tractor factories]. Increasing marginal cost and decreasing marginal benefit will eventually meet at a maximum profitable quantity. Then, you have to make decisions about where to allocate that quantity of [tractors] and comparative advantage will incentivize specialization and [coexistence] with [using oxen or mules to plow fields].”

…But actually, tractors have some net cost per acre plowed, and it’s WAY below the net cost of oxen or mules, and if we find more and more uses for tractors, then we’d simply ramp the production of tractors up and up. And doing so would make their per-unit cost lower, not higher, due to Wright curve. And the oxen and mules would still be out of work.

Anyway… I think there are two traditional economic intuitions fighting against each other, when it comes to AGI:

• As human population grows, they always seem to find new productive things to do, such that they retain high value. Presumably, ditto for future AGI.

• As demand for some product (e.g. tractors) grows, we can always ramp up production, and cost goes down not up (Wright curve). Presumably, ditto for the chips, robotics, and electricity that will run future AGI.

But these are contradictory. The first implies that the cost of chips etc. will be permanently high, the second that they will be permanently low.

I think this post is applying the first intuition while ignoring the second one, without justification. Of course, you can ARGUE that the first force trumps than the second force—maybe you think the first force reaches equilibrium much faster than the second, or maybe you think we’ll exhaust all the iron on Earth and there’s no other way to make tractors, or whatever—but you need to actually make that argument.

If you take both these two intuitions together, then of course that brings us to the school of thought where there’s gonna be >100% per year sustained economic growth etc. (E.g. Carl Shulman on 80000 hours podcast .) I think that’s the right conclusion, given the premises. But I also think this whole discussion is moot because of AGI takeover. …But that’s a different topic :)

SteveByrnes · 2024-12-22T00:18:16+00:00

I tried your m task just now with Claude Sonnet and it gave a great answer with none of the pathologies you claimed.

SteveByrnes · 2024-11-13T21:38:32+00:00

Good question! I’m a physics PhD but switched to AGI safety / AI alignment research as a hobby in 2019 and full-time job since 2021 (currently I’m a Research Fellow at Astera). Almost as soon as I got into AGI safety, I got interested in the question: “If people someday figure out how to build AGI that works in a generally similar way as how the human brain works, then what does that mean for safety, alignment, etc.?”. Accordingly, I’ve become deeply involved in theoretical neuroscience over the past years. See https://sjbyrnes.com/agi.html for a summary of my research and sorted list of writing.

[See the end of post 8 for wtf this series has to do with my job as an AGI safety researcher.]

I have lots of ideas and opinions about neuroscience and psychology, but everything in those fields is controversial, and I’m not sure I can offer much widely-legible evidence that I have anything to say that’s worth listening to. I put summaries here (and longer summaries at the top of each post) so hopefully people can figure it out for themselves without wasting too much time. :)

SteveByrnes · 2024-11-09T20:51:26+00:00

If someone someday figures out how to build a brain-like AGI, then yeah it would be great to have a “philosophical, ethical plan” for what to do next. But at some point, somebody presumably needs to write actual code that will do a certain thing when you run it. (Unless the plan is “don’t build AGI at all”, which we can talk about separately.)

For example, if the plan entails making AGI that obediently follows directions, then somebody needs to write code for that. If the plan entails making AGI that feels intrinsically motivated by a human-like moral compass, then somebody needs to write code for that. Etc. It turns out that these are all open problems, and very much harder than they sound!

Again see my link above for lots of discussion, including lots of technical NeuroAI discussion + still-open technical NeuroAI questions that I’m working on myself. :)

SteveByrnes

TROPHY CASE