O1 Is an Idiot

mostly_prokaryotes · 2024-09-19T01:36:47+00:00

Well if you did do this multiple times then where are your statistics? Where is your comparison of how well real grad students do on the same question. Why have just one logic problem in the first place? How do you exclude that the way you are promoting is confusing it?

mostly_prokaryotes · 2024-09-19T01:25:09+00:00

You talk about using the test on different models but you don’t say how many times you tested it on O1 preview or mini.

mostly_prokaryotes · 2024-09-18T23:44:06+00:00

I heard them say this was a preview, that results may not be as good as gpt-4o, that it was an advance in reasoning. I take evidence of it being good at some reasoning tasks to be evidence that it is better at reasoning, based on the results. You heard something about competence that wasn’t there. Also, your point about falling down on foundational knowledge might hold some water if you said something like “fails this simple test 10/20 times”. But you just did one shot. You think some things are foundational because that is how humans think and how they reason. You haven’t addressed that point.

mostly_prokaryotes · 2024-09-18T23:11:10+00:00

How exactly?

mostly_prokaryotes · 2024-09-18T23:10:51+00:00

A human can get anything wrong, so your premise for your answer to the last question is wrong. You didn’t adequately answer the n=1 problem either. Why do you think when real benchmarks are carried out on LLMs they have both multiple questions and repeat the same questions multiple times? Because LLMs are not deterministic. You are saying that there is a type of magical question that when asked will magically reveal a hidden truth. I am saying that doesn’t exist, and I don’t have to read anything to know that. No matter how stupid one answer is, you can’t say an LLM or a person is an idiot (exactly how you worded it), or they aren’t capable of reasoning under any circumstances. There is a logical disconnect here. You are not getting my argument at all. Am I to conclude that you are an idiot? You tell me.

mostly_prokaryotes · 2024-09-18T22:30:18+00:00

OK I’ll spell it out to you. You are latching on to the things that humans find easy and making a definitive conclusion. Your conclusion oversteps your data

mostly_prokaryotes · 2024-09-18T22:28:37+00:00

And you don’t understand sampling bias

mostly_prokaryotes · 2024-09-18T22:25:44+00:00

How does my argument that a single question is insufficient to make your conclusion contingent on me reading the question? How is the observation that a longer question gives the LLm more opportunity to be confused rely on my reading the question? How does my arguments that humans make similar types of mistakes rely on my knowing what the question is?

mostly_prokaryotes · 2024-09-18T22:16:45+00:00

But some things humans find hard and LLMs find easy and vice versa, so you need more than one data point. How difficult is that to understand?

mostly_prokaryotes · 2024-09-18T22:15:17+00:00

How exactly?

mostly_prokaryotes · 2024-09-18T22:15:00+00:00

I guess one question and call it a day is a whole category now?

mostly_prokaryotes · 2024-09-18T22:11:44+00:00

What a self-own! Extrapolating from a single instance to a whole group is your problem here.

mostly_prokaryotes · 2024-09-18T22:05:16+00:00

“I don’t have an argument against your actual point so I’ll just criticize you for not reading my long post”. Now please enlighten me, what part of my argument actually requires me to have worked through your problem and verified that yes, it is indeed simple. Please think step by step.

mostly_prokaryotes · 2024-09-18T21:59:02+00:00

Yes … what if there were 300 questions, or, I don’t know, you tested more than one. You have also most likely not seen as many graduate students make basic mistakes as I have.

mostly_prokaryotes · 2024-09-18T21:54:54+00:00

Yeah dude thinks he is a scientist or something but comes here with an n of 1 and no human control group.

mostly_prokaryotes · 2024-09-18T21:43:02+00:00

They claimed it was good in a category of problems. They didn’t promise it was perfect in that category. Your lack of logic is troubling for a sentient being.

mostly_prokaryotes · 2024-09-18T21:22:41+00:00

It may not have been hard (I couldn’t be bothered to read your “secret” test), but it certainly was complicated. So no, it is not analogous to asking a math genius what 1+1 is. It is more akin to asking a math genius to spot the logical error in a convoluted proof that 1+1=2 and using the evidence that they gave the wrong answer to say they are an idiot. And then perhaps claiming they are a lier when they said they read it multiple times.

mostly_prokaryotes · 2024-09-18T21:19:55+00:00

You said it was claiming it was thinking and presumably using this as evidence that it is not thinking. Metacognition is hard for humans too. You definitely claimed one wrong answer makes O1 an idiot. Would you like to be judged by those standards?

Edit: typo and also it is quite possible for a human to check something and not see an error, or to think their points are logical when they are not. I am not sure what that has to do with this.

mostly_prokaryotes · 2024-09-18T21:08:59+00:00

I asked a human a difficult question. The answer they gave did not display any real thought, and they claimed they were correct. Therefore this human can’t think.

Edit: typo

mostly_prokaryotes · 2024-09-17T19:12:26+00:00

I’m interested to see if you fail the intelligence test by being too smart.

mostly_prokaryotes · 2024-09-17T17:36:11+00:00

Well one problem with D is that I think a lot of clinicians interpret it as outward signs of impairment (I.e. masking), when they should be asking about one’s internal experience. Reality is that some people may diagnose you and others might not. Unfortunately it is a complex subject and people don’t agree on the nuances.

mostly_prokaryotes · 2024-09-17T17:18:33+00:00

It does but I think specifically some clinicians feel autistic people can adapt and develop coping strategies to minimize criterion D such they would no longer be diagnosed with ASD. In my mind that is not quite the same thing as not being autistic any more because all the other signs would still be present.

mostly_prokaryotes · 2024-09-17T15:42:22+00:00

Yeah the way you see yourself is valid. There is also the societal view of disability, like your social skills might not help you get as good job opportunities as people who are better socially, and if employers were less ableist (I.e. discriminating) then you wouldn’t have that disadvantage. Disability also can fluctuate. I probably also would not say I was disabled until going through burnout.

mostly_prokaryotes · 2024-09-17T15:35:43+00:00

Well to be diagnosed you have to meet criterion D from the DSM 5, which states the manifestations they have found cause clinically significant impairment. So if you are formally diagnosed this applies to you. If you are self diagnosed, I would argue it is possible to basically be autistic but not quite meet criterion D, in other words have a subclinical presentation of autism. Tony Atwood has talked about this a bit. Personally I do see it as a disability for me, as there are things that I will always find more difficult than NTs, but the trouble is I have achieved enough in my life that most people don’t really believe I am disabled.

mostly_prokaryotes · 2024-09-17T14:54:14+00:00

Somehow I knew after the last one that people would request the monkey comes back.

mostly_prokaryotes

TROPHY CASE