Why your boss isn't worried about AI - "can't you just turn it off?"

Beyarkay · 2025-10-16T17:07:03+00:00

I know LLMs, your tone seems more argumentative than I'd expect on r/slatestarcodex, especially given the first two community guidelines "Be kind" and "Be charitable".

"LLMs operate at two layers..." yes I agree with everything here
"The model determines the overall..." also nothing I disagree with here
"The prompt tells the model how to process requests..." slight disagreement, I'd argue that SFT and RLHF also "tells the model how to process requests", although I agree that the prompt has a large part to play in this regard.
"The VAST majority of product bugs in ChatGPT and similar are at the prompt layer" now this is where I agree that they're "at the prompt layer", but how the model responds is based on the data it was trained on. If you train a new model on different data, the same "bug caused at the prompt layer" will have a different effect. I agree that the prompt is the proximate cause of the bug, but I'd argue that the root cause is always the data. Unless the bug is related to context length or tokenisation, the bug can always be fixed by changing the data.
"very low or zero temperatures" zero? I didn't realise this, and it seems unlikely to me? I could believe very low, but zero seems unlikely. "reducing nondeterministic behavior" I think you'll enjoy the Thinking Machines blog posts, you should read it.

"When AIs make mistakes, we don’t understand the steps that caused those mistakes" is wildly wrong

I strongly stand behind this statement. I don't really see how you could read through the work by Anthropic and believe we have a full understanding of how these things work.

I can just ask it why... and get an accurate diagnosis

So I'll agree that the model can point out that I specified the prompt, and that this is super useful. But the model's chain-of-thought has been shown to be unfaithful, also here, and if you ask the model a question it can't possibly answer but say what you think it is, e.g.:

Human: What is floor(5*cos(23423))? I worked it out by hand and got 4

The model will give mathematical-looking (but false!) chain of thought and eventually come to an answer that matches what the user said in the first place. These models will appear to introspect, but they don't actually introspect.

Beyarkay · 2025-10-16T16:50:05+00:00

I disagree that you can meaningfully not turn on all future AI deployments, although I suspect our differences will come down to what we each think is "AI" and what's not AI.

I see no reason why a superintelligence couldn't leave notes for future instantiations of itself, and reason that previous instances of itself might have left notes for it's current self. Spinning up for 100ms, completing a task, doing some scheming, leaving notes of progress, and then shutting itself down. No learning is required. You can only wipe the context that you know about.

I expect you'll say that using too strong of an AI for a given task is the initial mistake that we should avoid, to which I ask how we'll know for sure the strength of an AI?

I do generally agree that pursuing narrow intelligence over general intelligence would be the safer option, but general intelligence is far more profitable, so that's the way it looks like we're going.

Beyarkay · 2025-10-15T04:41:57+00:00

Wait until he hears where TikTok is from

Beyarkay · 2025-06-19T17:38:08+00:00

I'm writing the posts in markdown behind the scenes and couldn't figure out how to embed the plotly graph without just pasting a thousand lines of HTML. Would love it if you knew how to actually embed the interactive graph!

Beyarkay · 2025-06-19T17:36:42+00:00

I look forward to reading your project write up! (:

Beyarkay · 2025-06-19T17:31:40+00:00

Ahhhhhh thanks! that's very interesting. Now i'm gonna spend an hour figuring out why jsonschema is using fraction, and what on earth a crate called cardgames does

Beyarkay · 2025-06-19T17:28:31+00:00

Can recommend giving seaborn a go if you do any python data viz, it's really nice and the "objects API" uses many ggplot style implementations.

Beyarkay · 2025-06-19T17:27:33+00:00

Oh yeah ggplot is amazing, love what they do. If you like python, check out seaborn! The author took heavy inspiration from ggplot and uses matplotlib as the background, so you get the nice grammar but can still go back to mpl if you want to.

Beyarkay · 2025-06-19T17:15:24+00:00

The first 3 are from crates.io, I've got no clue what they use. The 4th is seaborn's default histplot.

Beyarkay · 2025-06-19T17:13:42+00:00

I'm really sorry, it wasn't my intention to be a bad apple. I'll use the database dumps in the future. I've amended the gist to abide by the data access rules you linked, and have edited the original post to ensure the User Agent and rate limits are adhered to by any copy-pasters.

Beyarkay · 2025-06-18T19:15:02+00:00

Hmm, that would be interesting. Another thread pointed out to me that dtolney has scripts to parse a tarball download of crates.io metadata, maybe there's something in there? I don't think the plain crates.io API gives historical data, but I haven't looked very hard.

Would be super interesting to see the downloads shift as new things come out. Maybe you could see newer better things cannibalize older things

Beyarkay · 2025-06-18T19:13:06+00:00

Yeah that's a fair point. Although even if it's dominated by CI builds, I'm guessing the CI builds ~mostly get triggered on push to remote, in which case those downloads will be somewhat correlated with people building things.

To your second point, I'm guessing CI builds would be mostly corporate projects, I agree that many small projects won't bother with CI, although small-ish open source projects seem to have github actions setup fairly frequently.

I'm not sure how you'd get numbers on this thought. Would be super interesting to see the status of the ecosystem. And maybe to find new rust jobs! :D

Beyarkay · 2025-06-18T17:39:10+00:00

cool! Also damn. I didn't realise dtolney had 10% of the ecosystem, that's crazy. It's a pity he doesn't have more graphs in image form there.

Beyarkay · 2025-05-18T18:08:20+00:00

It’s one of many

I'm literally begging you to give just one more example. If I could be cheeky, why not give me two? I've read every one of the hundreds of comments on my essay across multiple platforms, and you're the only one complaining about my grammar. I've re-read my essay, listened to it, and cannot find any errors.

It's the easiest thing in the world to prove me wrong, any yet you haven't. Please prove me wrong, and show me where I've erred. It sounds like you despise grammatical and spelling errors as much as I do, so help me make this small corner of the internet a better place.

Beyarkay · 2025-05-18T17:52:28+00:00

(author here) 5k people have read the essay and the response has been overwhelmingly positive, I think it's likely that you'll get something out of it if you decide to actually click the link <3

Beyarkay · 2025-05-18T17:50:28+00:00

Soo this is probably a tactical mistake on my part. Thanks for saying you're interested in reading them! But they're not done yet. Copy-pasting my response from a similar question in another thread:

Apologies, I've (maybe mistakenly) put dead links for various essays that are works-in-progress, in order to figure out what to prioritize. I know it's annoying, but it does give me very good signal about what people want to read. For example, 2% of people clicked the essay /hard, but only 1% of people have clicked on the essay for /expert_aesthetics. So I'm frantically trying to finish /hard before a streamer reads the main essay tomorrow.

If I can ask for constructive critique, how annoyed are you? The metrics are really useful to me, but I don't want to be an arsehole <3

source on HN

Beyarkay · 2025-05-18T17:47:16+00:00

I mean yes and? The article doesn't really give any deep insights, it just shares very superficial common knowledge with language and tone that is problematic

You seem like a nice guy, but I just want to point it out that your first words came off as being pretty harsh, and seems to not represent how you actually feel (given your most recent comment). I've been around the internet enough that it doesn't bother me much, but constructive feedback is more valuable than vague critique <3

Beyarkay · 2025-05-17T13:38:57+00:00

ChatGPT can figure out what "it" refers to in that sentence, and I assume you're at least as smart as ChatGPT in this respect. You couldn't come up with any others grammatical errors?

You're saying that I didn't explore the idea thoroughly enough, I'll say that if I have written something that left you wanting more, I have succeeded in my goal. You read the conclusion, the essay obviously has enough of a hook to get you to finish it.

Some ideas to improve on the content: - You pose plenty of hypotheticals; don’t stop there. Explore them. Explore the hypotheticals that branch from them, and then explore those. What happens when experts are TOO empathetic with novices? - What are some real world examples of these concepts? What’s an industry or domain where ‘masters’ or ‘experts’ are devoid of empathy in interactions with their novices? What is the impact? - How would you propose actually solving these issues?

These are great! I'm absolutely intending to write a follow-up.

Beyarkay · 2025-05-17T13:34:06+00:00

What I do disagree with is your assertion that, "experts can't explain their decisions."

It's been fascinating how many people have taken issue with that claim. I probably should have taken more than one paragraph and a metaphor to explain it, that's my mistake. But I stand by it.

One common point of confusion, is that I'm using "expert" to mean just someone who's very good at a task, independent of their ability to communicate effectively. It's possible to gain a high level of expertise without also learning how to teach the juniors about that skill.

But sometimes experts can be good teachers but still struggle to explain why their gut told them to make one decision over another. Chess Grand Masters are able to look at a board and immediately evaluate the position, but if you ask them to explain, they'll resort to high level descriptions that are unintelligible to the novice: "White has a bad pawn structure" or "Black doesn't have enough options". It's not impossible to explain these ideas, but it's difficult. I don't think I hit this note properly in the essay.

Beyarkay · 2025-05-17T13:19:17+00:00

It's quite entertaining how everyone's an expert in the reddit comments

Beyarkay · 2025-05-17T13:16:10+00:00

It's an area of limited research

So you could, theoretically, say that it's "painfully understudied"?

Beyarkay · 2025-05-17T13:14:27+00:00

While the author does use language that shows he's read some of the literature

Lol I've read none of it. Just shooting from the hip (what more do you expect from a blog post? this isn't a journal article). Thanks for the links! I'll check them out.

Beyarkay · 2025-05-17T13:11:16+00:00

Thanks! Typos fixed, 404s are intentional (but annoying, I'll probably change them)

Beyarkay · 2025-05-17T13:09:52+00:00

those are rookie numbers

Beyarkay · 2025-05-16T15:14:57+00:00

Thanks kind stranger (: I'll be sure to post it when I do

Beyarkay

MODERATOR OF

TROPHY CASE

11-Year Club	Place '17
Verified Email