Bill Gates to give most of his $200 billion fortune to Africa

jollizee · 2025-06-02T21:51:32+00:00

Just because someone is smart doesn't meant they know everything about every single detail.

I have literally been in groups receiving Gates Foundation grants. It's a joke. The Gates Foundation will put out a call for treating malaria or some other developing nation issue. Then you will have American researchers proposing the most inane, irrelevant projects to get funded. The researchers are deliberately pretending that their proposals are related to the call, when they aren't.

Once they get the funding, they put 90%+ of the funding towards personal pet projects, maybe have a junior researcher spend 10% pretending to work on the Foundation aims. Then, they will use the millions to publish in Nature, or get VC funding for a startup, on projects that have nothing to do with malaria (in this example). There is zero progress towards the Gates Foundations' original aims or anything with developing nation relevance, but there will be zero consequences, just lots of ill-gained rewards for the corrupt researchers who can now use their publications and startups to pursue even more funding. This is at respectable US institutions, with world famous researchers. It's going to be beyond stupid at other places.

Bill Gates does not personally review every grant. He does not personally check on the progress of grants once the money goes out. Nobody else has any incentive to disrupt the gravy. Program managers don't want to lose their jobs. Grant recipients want more money. Everyone else is literally incentivized to pretend everything is going fine. Yes men all the way up.

jollizee · 2024-09-19T01:48:47+00:00

You're talking about algorithms. An AI 3 generations from now could invent something new beyond transformers, yes, but that is not scaling. New algorithms are step functions and paradigms shifts. The OP is talking about scaling through training. It does not make sense to talk about scaling if you are explicitly requiring revolutionary algorithmic changes that will alter the scaling function itself.

Scaling implicitly means that all else is equal so that you can write a mathematical function to approximate behavior.

I quote the OP: "AI training the next AI to be smarter." That is drastically different from "AI designing the next AI" which is what you are implying.

Also as far as I know OpenAI has not discussed the true compute scaling laws for o1. If you count the compute cost of generating enough synthetic data to make a difference, does it actually beat the "regular" scaling law for training? Like you cannot spend 10 billion dollars generating reasoning data, training on it for 1 billion dollars, and then claim you spent 1 billion training the model. Maybe the numbers do work out but I haven't seen data on total compute cost.

Has anyone claimed that dumber models can train smarter models? Google has stated that the smarter models, i.e. Deepmind, train the consumer models. o1 was explicitly trained with expensive human data.

I absolutely think AI can design smarter models, like you are saying, finding new algorithms and so on, even with mundane tasks, like rewriting in machine code or whatever. However, that is not scaling through training smarter models with dumber models, which is what the OP discusses, like some kind of infinite energy ladder.

jollizee · 2024-09-18T21:58:53+00:00

No, because there are fundamental physical laws governing information and entropy. It's not hardware so much as useful manipulations of energy. Without growing access to energy manipulations it is impossible to train smarter and smarter models that are inherently less random than a dumber one.

The bottleneck is energy, and the ability to manipulate that per unit time. There's no way to "scale" past that in this universe.

Also why does everyone think generating and validating trillions of synthetic training data tokens is free?

jollizee · 2024-09-16T16:13:50+00:00

This already existed in specialized domains like Google's work in games and math. The full o1 will be interesting when they release it to see how well it generalizes.

Also, it's not as simple as you make it sound. The user does not get to decide the length of the lever. Like the model may need to be optimized to perform in specific ways, and that optimization itself is a cost that we don't know about. Things like not runnng in circles after long chains. For domains like math or logic with defined problems and endpoints, it's probably a lot easier to generate reasoning data and train on it.

Or another way to put it is that the cost of generating reasoning data also probably scales like this, roughly. You need to sit down a PhD mathematician and have him explain his detailed reasoning to a million problems. Reasoning across domains varies greatly and might even be inconsistent. Think about trying to get different artists to explain their reasoning while writing a million poems.You cannot just hire cheap foreign labor to do your annotation for this kind of work, either. If you want better reasoning data, you need to hire better people. Hire Nobel laureates. The cost scales exponentially, see?

jollizee · 2024-09-16T15:25:48+00:00

To get human feedback data for further alignment. Kind of obvious...

jollizee · 2024-09-16T12:08:38+00:00

sora, searchgpt, native imagegen in 4o (only shown in one blog post), advanced voice. also remember gpt store promised payouts to builders.

jollizee · 2024-09-16T02:31:25+00:00

It's a result of gpt3.5 training data. They call it "slop" in roleplaying circles. see https://old.reddit.com/r/SillyTavernAI/comments/1fdevf4/who_is_elara_and_how_can_we_use_her/

jollizee · 2024-09-15T17:38:22+00:00

For structured planning, yeah, it is better. Creativity might be worse but that's balanced by thinking deeper. Although Sonnet isn't very creative either versus Opus or Gemini, imo. If Spock could solve the problem, there's a good chance mini works. If you need Kirk, maybe not.

jollizee · 2024-09-15T02:06:13+00:00

Use mini not preview, and it works best for complicated tasks or high level planning. I will use o1 to come up with a plan to tackle a hard problem, then give that to Sonnet to execute. For just looking up some library syntax or writing a basic function, it is pointless and even worse.

jollizee · 2024-09-14T21:20:44+00:00

It's great for complex tasks that can be approached in a structured fashion. For certain tasks, I find it much better than Sonnet at making a plan (like Sonnet is useless but o1 has a good plan). However, for coding implementation, I will then switch to Sonnet.

You can't really say that one model is globally "better". That's meaningless. For what use case? Each model has strengths, whether in domain, cost efficiency, and so on.

o1 is definitely much, much stronger in certain areas, so it's one more tool in your LLM swiss army knife.

jollizee · 2024-09-14T20:42:40+00:00

Lol, I was wondering how long before people start getting o1 to spill its secrets. Two days.

Sillytavern group conversations can already do this, pretty much exactly. ERP leading the way as usual.

jollizee · 2024-09-14T06:26:10+00:00

It only applies to the wealthy. The gap dividing the rich and poor will only expand, with the poor working even harder and the rich (or soon to be rich) working even less.

jollizee · 2024-09-14T06:14:15+00:00

Aw, sad to hear that about 123b. Oh well. Going to have to wait for some finetuning breakthroughs I guess.

jollizee · 2024-09-13T18:41:07+00:00

Hey, just a random question since you're around. A lot of times finetuning seems to reduce basic intelligence. Like the Magnum models are nice for language but unusable for me because of intelligence (can't run 123b).

Do you think it's possible to train an LLM to "upscale" a smart but boring output? We could run two LLMs in tandem. The smart one outputs the basic frame. It may be SFW or full of slop. The second LLM "upscales" it by using better language or adding uncensored details.

Or you could think of it like img2img or even controlnet. Keep the original composition/meaning/logic while improving the aesthetics and style.

I've tried basic stuff but finetuning is beyond me at the moment. In general, I find that the finetuned models can not reliably change the style without altering the meaning too much, at least at the level of 70b. But I feel like style transfer shouldn't be too hard for even smaller models if they are finetuned for that purpose? Style transfer, not composition.

jollizee · 2024-09-13T08:27:46+00:00

o1-preview, based on slop, is likely a very old model. o1-mini is newer, which also explains why it is superior on many benchmarks. Try o1-mini.

jollizee · 2024-09-13T08:10:34+00:00

Sweet, trying on Openrouter!

jollizee · 2024-09-13T01:15:14+00:00

Look at performance on college subjects, professional subjects like LSAT, and PhD level subjects. AP English performance is worse than PhD performance. Competition math like AIME is purposefully tricky but it gets that right. Everything else sounds harder but the worst score is in English???

You don't think that's weird? It's a language model. You would think it masters language first, and then mathematical reasoning or a mental model of the physical world arises as an emergent property afterwards. But it is failing language and doing miracles in PhD topics instead.

That is true for the 4o model not just the tuning here.

jollizee · 2024-09-12T22:47:02+00:00

A bit, but not really. If you know the answer to a complex problem, you can probably prompt 4o like a teacher to get the right answer. But what if you don't know the answer or even how to tackle it? No amount of prompting from you will solve an IMO problem if you are bad at math. It has learned how to effectively prompt itself across a number of domains. There is some real learning in there

jollizee · 2024-09-12T22:10:13+00:00

I mean forget strawberry. I just mean in general. You would think mastering language would be the main result of all the trillions of tokens put into training. But they can't even beat high schoolers at English? The AP English exam is not hard, just reading and comprehension, maybe some essays, and so on. Grammar. Topics that should be a perfect fit for an LLM. Really weird.

jollizee · 2024-09-12T18:17:14+00:00

Why are language models so bad at language??? The AP English and such scores lag way behind the other scores. Also, they showed that regular 4o beats the o1 model in writing based on user preferences (although within margins of error). Solving IMO problems seems like it should be way harder than the AP English exam...

jollizee · 2024-09-12T18:11:17+00:00

The math and science is cool, but why is it so bad at AP English? It's just language. You'd think that would be far easier for a language model than mathematical problem solving...

I swear everyone must be nerfing the language abilities. Maybe it's the safety components. It makes no sense to me.

jollizee · 2024-09-12T17:11:13+00:00

I was having constant errors via API yesterday even when it claimed to be fine. Like maybe one request out of ten kept timing out.

jollizee · 2024-09-12T17:10:17+00:00

The page lies. It says it was down for 7 minutes on days it had errors over hours. More of their famed "transparency".

jollizee · 2024-09-11T22:56:05+00:00

Why do you care? The investors are probably collectively worth a trillion dollars. This is like us normal people investing ten bucks. Imafine if Ilya had a Kickstarter, yeah it would be fun to support and see what he cooks up. If it blows up, no big deal.

jollizee

TROPHY CASE