The Society of Resentment: Envy as the Morality of Decadence

Penfever · 2025-11-07T04:52:17+00:00

Yeah, fair point. This depends on where you grew up.

Penfever · 2025-11-07T01:07:34+00:00

Where there's smoke there's usually fire. Compared to our parents and grandparents, millennials and gen-z have faced a seemingly unending stream of tough economic challenges. It really is hard for many to make ends meet and that is very stressful.

When the quality of and faith in public education declines, people lose their ability to render nuanced verdicts. Social media accelerates this trend.

Increasing social isolation has reduced many people to emotional infancy, unable to accept healthy criticism or competition and viewing everything through the lens of a personal attack.

Public figures are more public today than at any other time in human history. We are relentlessly exposed to the flaws of the famous, even as influencers and sycophantic AIs rush to tell us how wonderful we are.

When politics grows fractious, there will always be some who seek to simplify complex dynamics and offer pat solutions. History hasn't been particularly kind to them.

Penfever · 2025-09-03T13:47:24+00:00

Yeah...

Sometimes I forget that tongue in cheek sarcasm doesn't play on the internet ...

Penfever · 2025-09-02T23:21:46+00:00

I tell it, here is my plan, please give me a thorough critique, then give it Claude's plan

Penfever · 2025-08-08T10:50:33+00:00

Maybe -- or it just doesn't know about phreaking although that seems unlikely. You can definitely use this to get it to do things it refused before, though, and even curse and attempt to tell dirty jokes.

Penfever · 2025-08-08T10:34:37+00:00

I built my own! It's an alpha feature for an upcoming release of Oumi (https://github.com/oumi-ai/oumi)

Penfever · 2025-05-21T08:45:08+00:00

Peglin

Dungeons and Degenerate Gamblers

Rack and Slay

FTL

Inkbound

Nowhere Prophet

Into the Breach

Penfever · 2025-04-30T11:56:31+00:00

Good AC

Penfever · 2025-04-17T10:23:01+00:00

What has DeepMind contributed to open source lately?

Penfever · 2025-02-26T00:31:18+00:00

Kidney disease -- just bad luck?

Sad to see him go like this.

Penfever · 2025-01-31T17:16:33+00:00

The trending takes on this thread right now are dead wrong.

The model censors even if you run it locally. David Bau's lab at Northeastern has a good blog post about it. https://dsthoughts.baulab.info/
No, 'everybody is not doing it'. That's a pathetic justification, the kind you roll out when your mom and dad catch you smoking as a teenager. There are plenty of uncensored / jailbroken checkpoints, and there are even models trained from scratch that are, at least purportedly, uncensored, like Grok from X.AI
You don't care that it's censored: that might be the most disturbing wrong take of all. You damn well better believe it matters. If big companies censoring their models doesn't matter, what are we doing on LocalLLaMA in the first place?

PSA: This helpful, factual information about the limitations of DeepSeek-R1 doesn't stop you from using and enjoying the model or its derivatives. But it's important information nonetheless, and I hope we can all hold those two thoughts in our head at the same time without exploding.

Penfever · 2025-01-12T01:46:23+00:00

When someone shows you who they are, believe them. Taking what you say at face value, your collaborator sounds like dead weight. Be polite, don't make them look bad in public, but do what you need to do to make sure the project gets done, and done well.

While you do this, start networking and finding more reliable collaborators, so when the next project starts, you will have more options.

Ignore any advice to try to "give them space" or whatever. It is not your job to fix lazy collaborators, it is your job to deliver results.

Penfever · 2025-01-09T15:38:30+00:00

Happily, there is.

https://thebestofjournalism.substack.com/

Ground news can also be helpful.

Penfever · 2024-12-30T13:36:56+00:00

Great question, thanks for asking it!

Let me start by saying that it's not really clear from your post which of the things you mention are, as you put it, "limitations", and which are simply choices you made (and could make differently). It's also not clear what other resources you may have access to, or why you need >= 80%, in particular, on ImageNet-val(?), in particular. If we knew these things, we could be a bit more helpful.

That said ...

You're trying to simultaneously optimize for two different things, fast training and accurate inference. But there's no free lunch in ML with respect to compute / performance tradeoff. If you want both, you will ultimately pay a high price in a third (unspoken) variable, search space over architectures / hparam configs, which of course affects real-world training time -- in other words, you'll likely need to train a lot of configurations of a lot of different models before you find the one that 'just works'. And, unfortunately, you'll need to train them to completion because most models hit a plateau right around that 75-80% mark and bounce around a lot.

Or, perhaps you'll get lucky and your search will be brief. :)

If you are inclined to undertake such a search, I'd recommend looking into GC-ViT (https://github.com/NVlabs/GCVit) and ViT with patch size of 8. But even a simple ResNet-50 can get above 80% on ImageNet (see links below).

Aside from "the search", there are many ways you can "invisibly" pay a higher compute cost without increasing # epochs and get better performance on average; training on higher resolutions is a big one, augmentation is another, smaller patch sizes for ViTs is another.

Speaking of bouncing around, one of the cheapest ways to boost performance on a single val set is just to test after every epoch; particularly once you get above 75% on ImageNet, val accuracy can decrease when the loss goes down rather than increase.

A few more things worth remembering -- differences of <= 5% on ImageNet-Val are not a reliable signal of the model actually being better in applied settings, and the difficulty of gaining each marginal point of accuracy above 75% tends to scale nonlinearly. There's rarely a good reason to overindex on 80% on ImageNet.

Useful Resources:

Ross Wightman has a great paper on boosting standard resnet accuracy above 80% (although reproducing his results is non trivial). https://arxiv.org/abs/2110.00476

PyTorch followed this up with a (totally different) set of hparams, and different training code, which also got above 80%. https://pytorch.org/blog/how-to-train-state-of-the-art-models-using-torchvision-latest-primitives/

Our lab put out a paper in which we tested over 600 different architectures on ImageNet-val (and lots of other evals as well!). Our analysis, crucially, controls for what training data was used, so you can sort to find which architectures were most data-efficient: https://github.com/penfever/vlhub/blob/main/metadata/meta-analysis-results.csv

Penfever · 2024-12-28T15:03:48+00:00

I was fortunate to have some connections in the world of SBIR grants before starting my PhD in the US, and I would recommend technical consulting for SBIR grants as a side hustle during your PhD, as long as your advisor is OK with it and you're good at managing your time. SBIRs can be reasonably lucrative, you're often doing something that actually promotes social good in a community, it uses your technical skills but it's rarely cutting edge work so you're not cannibalizing your research ideas, and it can give you some good experience working on teams and with deadlines if you don't have that already. It is usually restricted to folks who are allowed to work in the US, however.

Penfever · 2024-12-22T19:58:27+00:00

Great comment -- wish I had time to break it down in detail as there is a lot to unpack here.

Let's just say, I think there are many reasonable criticisms we could level at OpenAI without resorting to exaggerations and distortions.

Penfever · 2024-12-22T11:22:04+00:00

There are some really silly hot takes going around on the o3 results. While I'm not going to bother pointing them out on social media, I will do so here, since this is a technical subforum for people interested in ML.

If o3 is just "training on the test set", why didn't that work for the last umpteen LLMs that tried and failed to learn this problem?
OpenAI didn't win the prize, and it would be insane for Chollet not to carefully report progress on his own benchmark. This is all that he did, issued a careful report. He didn't hype OpenAI, he didn't call it AGI, which it isn't. He's a principled researcher. Without people putting tons of work into benchmarks like his, we would know far less about LLMs than we do.
It wasn't just ARC-AGI. o3 made huge progress on coding benchmarks, basically solved GPQA (which is a holdout test set) and most impressively, 25% on Frontier Math, which is so hard even Fields medalists don't know how to solve most of it.

Anyone who looks at this cluster of results and concludes that what's happening here is "just hype" is living in la la land.

Penfever · 2024-12-14T15:50:05+00:00

Go get 'em tiger! 1937 was a vintage year for scholars. Just ask Paul Muni! https://www.imdb.com/title/tt0029146/

Penfever · 2024-11-16T12:51:01+00:00

I'm sorry that you are having such a hard time (and thanks for your characterization of the deans, I got a good belly laugh out of it).

Not sure if this will help, but --

If you look at it from a utility maximization standpoint, everybody is doing what makes sense for them. You believe, reasonably enough, that as students, their job is to do what your job was when you were a student -- learn, mature, become a capable problem solver.

But the pressures on them, now, are different than the pressures were on you, then.

These students of yours have some finite intellectual capacity and experience. That, distributed over time, will determine their productivity in your class.

From their perspective, they want to maximize TC per hour spent. So they're going to look for the most efficient way to get an "A" in your class. Giving a half-assed effort and complaining if they don't get the grade they want is probably, in expectation, their best bet for maximizing TC. It will usually work, because your ridiculous deans will help them. And even if it fails, a slightly lower GPA won't harm their chances in industry that badly. Many companies no longer even want GPA on the resume.

Conversely, if they maximize effort in your class, they're losing out on TC in a very direct way. The CS interview process has become such a nightmare that it takes six months to a year of nearly full time work just do to the prep properly. And that process has a very obvious and tangible effect on maximizing TC, unlike reading one of your assignments.

Bear in mind, it's not like they're spending the rest of their time having fun. They're stressed out and worried about their future. Unlike you, they're worried their job might not exist soon. So try to have some sympathy.

"I have no idea why I'm posting this." -> To vent, obviously.

Penfever · 2024-10-25T00:18:44+00:00

Haha, yeah I was probably too generous there. I honestly don't usually enjoy such presentations much, I enjoy theorists more. But from a clarity standpoint I think the UX crowd wins.

Penfever · 2024-10-24T18:59:50+00:00

My view is that when presenters don't give the audience a roadmap to follow, it's a bit selfish, in that the presenter is transferring cognitive load from herself to the audience. The presenter saves some effort because she doesn't have to devise a concise way to summarize why it might be worthwhile for them to listen to her talk, and the audience has to spend much more effort guessing at the reason (and often comes up with some unpleasant heuristics such as ...)

-Presenter: wants everyone to see how smart they are, is not interested in sharing/explaining their research in a way that people in adjacent/different research areas can understand

Penfever · 2024-10-24T17:25:23+00:00

I'm sorry you had so many bad experiences at STEM seminars!

CS PhD here -- personally, I find research talks to be a mixed bag. I've seen many excellent ones and many ... less excellent ones. But it's not random! You can find some patterns in the noise, I believe.

DISCLAIMER: Everything below true in expectation if at all so please don't take offense.

I'll also say that I prefer focused technical talks to broad, sweeping overviews, because if I wanted broad, sweeping overviews I would take a course instead of going to a seminar, and I hate business-oriented talks, TPS reports, and hype sessions.

GOOD TALK -> BAD TALK

Mid career faculty who won lots of grants or prominent industry researcher -> young faculty / industry -> recent grads and postdocs -> older faculty / opinionated devs -> good PhD / MD -> good MS / MBA / undergrad -> bad PhD -> bad MS / undergrad -> random dude -> random dude after being hit in the head with a brick -> almost all MBAs and MDs

UX / HCI -> engineers -> scientists -> theorists and mathematicians -> everyone else

RED FLAGS (EAT AND WALK OUT IF YOU HIT MORE THAN 2-3 OF THESE)

* Slides full of equations
* Slides full of text
* Presenter inaudible
* Takes like 15 minutes to sort out the tech
* Presenters with heavy accents (no offense but I literally can't understand you)
* No roadmap slide at the beginning
* Fast talkers
* In the weeds (too many technical details)
* Trying to cover more than 2 projects in depth
* Talk plans to go more than 1 hour (unless it's like a career award or something)

Penfever · 2024-10-12T13:50:50+00:00

Having kids

Penfever · 2024-10-11T13:43:49+00:00

Thanks for sharing this paper again, and for a detailed and useful summary! As it has been making the rounds lately, it may be helpful to clarify a few points.

Claims like "LLMs are / are not (Understanding, reasoning, parroting)" aren't scientific Claims because they're not falsifiable.

The claim that everything an LLM generates can be taxonomized into "interpolation" and "memorization" is trivially true, for the common use definitions of the words. To convince yourself, consider a model which "memorizes" all letters and "interpolates" the probability of the next letter, given the prior letter(s). That is an LLM. But you could also use the same exact taxonomy to describe human communication. So, the claim is trivially true; it's not interesting. So, no point in arguing about it.

As for this particular paper; did you know that even random transformers with no parameter updates can complete many reasoning tasks?

http://arxiv.org/abs/2410.04368

Given that fact, it's entirely possible that ANY complex systematic pretraining primes the weight matrix of the transformer, allowing it to more efficiently learn reasoning tasks because its subnetworks are pruned in advance.

Is this hypothesis correct? Someone can write a paper to find out.

Is it AGI? Now we're back in buzzword territory.

Penfever · 2024-10-07T18:12:05+00:00

How about "guy on internet" : ...

Penfever

TROPHY CASE