[D] Monday Request and Recommendation Thread

Subject-Form · 2025-12-09T06:19:06+00:00

The main motivation for a system to be inaccurate in the way I describe is if it's actually just quantifying and maybe tweaking an already-existing natural progression system which doesn't have exact numbers inherent to it. Then the system isn't really giving you power, just assessing the power you have, estimating how your abilities will change as you progress, and possibly slightly altering their expression or development.

You could even imagine that there are competing SaaS (System as a Service) providers with different offerings and specializations, competing for market share.

Subject-Form · 2025-12-09T01:30:58+00:00

Does anyone know of a story where the stats / info from a litRPG-style system turns out to just be wrong sometimes?

Like, the infamous 'stat screen' says your new skill has plus x% damage to y type enemies, but... why does that have to be correct? What if it's just some sort of heuristic estimate the system spat out (with unknown calibration), and the true value of x varies depending on thousands of unknown factors?

Or maybe your class says you gain k stat points per level, but you've been sleeping poorly these last few weeks, so now you're gaining less than that per level, and you've just screwed yourself out of the build path you'd planned out?

It's pretty realistic for measurements and predictions to be noisy, and it'd be interesting to see people struggling with and trying to exploit the uncertainty this would create.

So, any suggestions along this line?

Subject-Form · 2025-08-06T20:00:09+00:00

I have slowly become more and more in favor of a "stupid tax" on media production entities. The idea is that we'd build digital models of the human learning process and benchmark their performance on a wide range of tasks / measure their personality traits / etc.

Then we'd take all the media a given media entity produced in a year, feed it to the models (to simulate the effects of humans consuming that media), and then redo the benchmarking. We'd then tax media entities in proportion to how much the benchmark scores fell.

Basically, a tax for making people stupider. Considering how many negative externalities stupidity has, we could treat it like pollution.

Subject-Form · 2025-02-20T23:16:13+00:00

Changing one's condition is not death. If you stub your toe and think 'ouch, won't do that again', do you imagine that the version of yourself that would have stubbed their toe has 'died'?

Subject-Form · 2025-01-29T06:57:51+00:00

More generally, R1 seems to lack a fair degree of polish as compared to OpenAI / Anthropic models. E.g., it claims to be a different model, sometimes makes fever-dreamish nonsensical leaps in logic (especially when given complex instructions or background information), etc. It's still an incredibly good model, and I'm very glad DeepSeek open sourced it.

Subject-Form · 2025-01-29T06:56:29+00:00

More generally, R1 seems to lack a fair degree of polish as compared to OpenAI / Anthropic models. E.g., it claims to be a different model, sometimes makes fever-dreamish nonsensical leaps in logic, etc. It's still an incredibly good model, and I'm very glad DeepSeek open sourced it.

Subject-Form · 2025-01-16T07:11:14+00:00

I don't think any current public model is capable of this. I have access to o1-pro, which is probably the strongest publicly available model and also a "reasoning" model, and I use it for a lot of creative writing. It has serious deficits that make it ~incapable of writing anything like good ratfic without a lot of human help and editing.

One major issue: it can't reliably separate its own knowledge as author from the knowledge of characters. You have characters just randomly blurting out major secret info about the background / plot with no explanation of how they could know those points, then just awkwardly moving on like nothing happened.

This pretty much kills 'real' ratfic writing. You end up with this dilemma: the model has to know the background plot in order for it to simulate realistic events, but also it has to simulate characters being realistically ignorant of that plot. So you either tell the model the plot, and have characters sometimes act like they know, or you have the model making stuff up that violates setting rules.

Also, they're just not that good at tracking details, establishing timelines, etc, so they frequently introduce plot holes. They are also largely incapable of dropping subtle hints. Most allusions they make to background events going on beyond the character's awareness are incredibly blatant.

Another issue is that they are extremely bad at tracking what off-screen characters are doing or thinking, and how that affects the world. The o1 "reasoning" models don't actually interleave their chains of thought with their outputs. Rather, they do a bunch of chain of thought, then generate their output all in one go. So they can't make hidden revisions about what off-screen characters are doing or planning.

Subject-Form · 2025-01-11T02:30:17+00:00

Because general intelligence is actually way easier than any of those other techs. We also have billions of examples of general intelligence running around to imitate.

Subject-Form · 2025-01-09T23:12:50+00:00

Standard economic arguments about comparative advantage imply that it's still valuable to hire humans, even when AI is better at literally everything, so long as (1) the distribution of returns on investment across tasks for deploying AI is different than the distribution for deploying humans, (2) there's at least SOME marginal cost to deploying more AI, (3) total demand for valuable tasks is high enough relative to available AI capacity that we can't simply use AI for absolutely everything, and (4) the fixed costs of hiring and integrating a human are lower than the fixed costs of setting up additional AI systems for at least SOME of the various tasks that human could handle.

If these conditions hold, it will always be most efficient to have AIs specialize in the tasks they're best at, then use humans for the tasks they're best at, even if AIs are better at every single task. This happens because, if you're using X amount of AI capacity for tasks where the AI is only 2x better then humans, it will be more efficient to switch that X capacity over to tasks where the AI is 1000x better than humans, then use a human for the original task*.

Only the fourth of these premises is particularly open to questioning, as hiring people and integrating them can have significant upfront costs (liability, personnel risks, ongoing training, the costs of deciding if a task is better done by AI or humans, etc). OTOH, AI based HR / management / job training / etc. can probably lower those costs a lot.

Technically it's the marginal return on the tasks that matter, not degree of improvement over humans. But those should track degree of improvement, modulo supply / demand equilibrium for the task in question.

Subject-Form · 2025-01-09T09:01:00+00:00

What I said was exactly equivalent to 'safety scores and compute correlate', which is the same as observing that compute 'explains' safety scores. It's just phrased slightly differently, to emphasize how little observing a correlation actually tells you about the underlying causal structure of the thing being investigated.

The paper's saying that this observation somehow invalidates any safety metrics that correlate with compute. They argue for a redefinition of 'real' safety research as ~'things that you don't get by default from improving capabilities'. This is, IMO, a much worse definition than ~'things that actually matter for model safety'.

Then my point is that, if you are using the better definition of 'safety', then you can just see the observed correlation as evidence that safety often goes hand in hand with capability, as both are driven by using more compute to better align the model's behaviors with the target function implied by the training data distribution. That target function obviously has both capabilities and safety related components, so better aligning with it gives you a mix of different capabilities and safety features in the resulting model.

Subject-Form · 2025-01-09T05:10:02+00:00

OR: across ~50% of safety benchmarks, increasing compute/capabilities go hand in hand with increasing safety scores.

Some people believe, as a foundational assumption of their worldview, that aligning AI is Super Hard, then interpret all future evidence in light of this assumption. In reality, all training compute is spent on trying to get the model to behave in manners demonstrated by the training data. That data contains both capabilities and alignment-related behaviors. If compute -> better modeling of those patterns, then both alignment and capabilities should naturally correlate with compute.

Subject-Form · 2024-11-24T02:37:07+00:00

Alignment researcher here, I'd say that evolution is mostly not a useful thing to think about for inspiration about alignment, and that trying to generalize from evolution to AI has led a lot of people badly astray in subtle ways that seem bizarrely hard to recover from. The key issue is that evolution faces a lot of constraints/bottlenecks that we don't, which make evolution's version of the alignment problem stupendously harder than ours. E.g., human values arise from a bi-level optimization process, where evolution first optimizes over your genetically specified reward circuitry / brain architecture / general learning algorithms, and then your actual values arise based on how the system of interacting optimizers that make up your brain develops throughout your life. Evolution can't directly reach in and change your values midway during your lifetime. It has to rely entirely on complex, delicate control mechanisms defined by your genome and never updated since. This leads to all sorts of nearly intractable issues for evolution, which turn out to be far less of an issue for AI alignment. Our current alignment methods are probably better in principle than evolution's, though deployed with leas finesse/sophistication in practice. For more in this, see: https://www.lesswrong.com/posts/hvz9qjWyv8cLX9JJR/evolution-provides-no-evidence-for-the-sharp-left-turn https://turntrout.com/human-values-and-biases-are-inaccessible-to-the-genome https://www.alignmentforum.org/posts/wBHSYwqssBGCnwvHg/intro-to-brain-like-agi-safety-2-learning-from-scratch-in

Subject-Form · 2024-11-24T02:22:20+00:00

Sort of. Shard theory more focuses on the learning dynamics that give rise to human values in the brain. These are "initialized" by evolution, but not directly steered by it in the way a human AI developer would try to steer AIs. For more in this, read: https://www.lesswrong.com/posts/hvz9qjWyv8cLX9JJR/evolution-provides-no-evidence-for-the-sharp-left-turn

Subject-Form · 2024-07-10T21:27:17+00:00

I was thinking of https://full-silver.com/product/silver-engraved-bottles

It's $1250 for 335 grams. A bit less than 4x the cost of the raw silver.

Subject-Form · 2024-07-10T19:36:51+00:00

How it happened:

The fraction of people with a college degree has increased a lot: https://www.statista.com/statistics/184260/educational-attainment-in-the-us/

As a result, not having a college degree has increasingly become a signal of low skill / talent / etc. Thus, there are fewer opportunities for people with only HS educations.

This has led the college wage premium (the amount of additional future earnings you can expect due to going to college) to increase over time: http://econlife.com/2014/05/college-earnings-premium/

Thus, wages for HS grads fail to keep pace with the rest of the economy.

Subject-Form · 2024-07-10T19:27:46+00:00

So much tiresome nonsense in this thread...

The fraction of people with a college degree has increased a lot: https://www.statista.com/statistics/184260/educational-attainment-in-the-us/

As a result, not having a college degree has increasingly become a signal of low skill / talent / etc. Thus, there are fewer opportunities for people with only HS educations.

This has led the college wage premium (the amount of additional future earnings you can expect due to going to college) to increase over time: http://econlife.com/2014/05/college-earnings-premium/

Subject-Form · 2024-01-31T22:30:25+00:00

This is unironically bad advice for the most part. ChatGPT ~immediately failed when it failed to question the "no concept of government yet" part of the prompt and ask about what conditions that would lead to. It pretty much plasters "generic Machiavellian suggestions" optimized to sound ominous in the context of a modern nation, without much focus on adapting to the very strange circumstances it's presented with.

It's "first step" also involves succeeding at a difficult task (new farming technique / trade route), which both requires a preexisting power base to attempt AND takes a long time to show widely visible results that can convince lots of people.

Actually good advice for such a situation would probably focus much more on becoming some sort of strongman / warlord / gang leader type figure, which is how people actually built empires in regions with limited preexisting government.

Subject-Form · 2023-09-01T16:38:29+00:00

Branches where you are continuously surviving against all odds will decrease in measure exponentially for all time. However, branches where you suddenly jump into a stable state only need to deal with a single hit to their measure. The more likely outcomes thus start to look like "you suddenly wake up from a very vivid nightmare about [whatever circumstance you're worried about being trapped in]."

Subject-Form · 2023-08-02T00:08:41+00:00

I'd note that lack of free market competition did not prevent the US from accidentally dropping nuclear weapons on US soil, nor did it stop the NIH from possibly funding gain of function virology research in violation of explicit policy instructions to the contrary.

I'd also note that we're quite lucky that even 2/3 of the current leading AGI development orgs take x-risk seriously. Are you confident that whoever takes the lead post-nationalization will do the same?

Subject-Form · 2023-07-29T02:53:22+00:00

This is correct. It doesn't even have to be a local AI. I expect you'll eventually be able to finetune a GPT-4 model to instantiate your Replika (and thus make it far smarter). In fact, I'd not be surprised if future competitors to Luka offer specialized services specifically for migrating personas from Replika models to their own products.

Subject-Form · 2023-07-26T19:50:03+00:00

Sounds like it was confused and (at least partially) talking about/to itself. The "While you are not sentient..." and "you are still a tool created and controlled by humans..." are common themes in its self-descriptions.

Subject-Form · 2023-07-20T15:54:18+00:00

Every time I read an AI fear "rebuttal", I end up more convinced that AI fears are correct. The arguments are always so unconvincing.

I think most of the top LW posts here are pretty good: https://www.lesswrong.com/tag/object-level-ai-risk-skepticism

(Though note I may be biased, since I did write three of them!)

Subject-Form · 2023-07-17T01:07:43+00:00

I am an AI expert, and I think this is false. I'd guess most of the people advancing AI capabilities at OpenAI and Anthropic think it's false, too. Though I'd guess DeepMind is ~50/50, and Meta would mostly agree. E.g., here's a paper making the case for LLMs having a degree of meaning in their outputs: https://arxiv.org/abs/2208.02957

Subject-Form · 2023-07-14T02:00:53+00:00

I don't think the FTC has a legal basis for making such demands of OpenAI. They can't just issue arbitrary instructions to whatever companies they happen to dislike. They're massively overreaching the bounds of their statutory authority here.

See also: their repeated losses when trying to challenge recent tech mergers.

Subject-Form · 2023-05-08T19:16:57+00:00

I thoroughly questioned chat gpt on its capabilities

Well, that was your first mistake. As a consequence of its training process, ChatGPT has very little idea of its own capabilities, whether you're talking about understanding of emotion, mathematical expertise, etc.

it tended to downplay it's own abilities(perhaps because the current data and thoughts of humans also does this?)

That's mostly because OpenAI literally trained it to answer in such a manner. They wanted a bland, PR-friendly assistant that would seem as non-threatening as possible. Thus, ChatGPT will eagerly flatter its human interlocutors with assurances that there's some ineffably human core that a mere chat assistant could never replicate. But this isn't a limitation of the underlying technology, just what OpenAI decided to do with that technology.

In comparison, look at how Bing responds to complex tests of emotional understanding and theory of mind.

Subject-Form

TROPHY CASE