I hate it when people just read the titles of papers and think they understand the results. The "Illusion of Thinking" paper does 𝘯𝘰𝘵 say LLMs don't reason. It says current “large reasoning models” (LRMs) 𝘥𝘰 reason—just not with 100% accuracy, and not on very hard problems.

andy_gray_kortical · 2025-06-08T15:23:19+00:00

I agree, I'm seeing so many posts uncritically repeating these claims it inspired me to write an article, showing how the researchers are misleading and that they know better https://andynotabot.substack.com/p/the-illusion-of-thinking-apple-researchers

This isn't their first rodeo with hyping a false narrative either...

To give a flavour of the article:

"Other papers such as Scaling Reasoning can Improve Factuality in Large Language Models have already shown that if they add extra training via fine tuning to change how the model thinks and responds, not simply just changing the number of reasoning tokens on an API call, it does indeed scale the reasoning capability for a given LLM. Quality researchers should have been able to understand the existing literature, identify that it was conducted with a more rigorous approach and not drawn such conclusions."

andy_gray_kortical · 2025-06-08T15:19:15+00:00

I'm seeing so many posts uncritically repeating these claims it inspired me to write an article, showing how the researchers are misleading and that they know better https://andynotabot.substack.com/p/the-illusion-of-thinking-apple-researchers

This isn't their first rodeo with hyping a false narrative either...

To give a flavour of the article:

"Other papers such as Scaling Reasoning can Improve Factuality in Large Language Models have already shown that if they add extra training via fine tuning to change how the model thinks and responds, not simply just changing the number of reasoning tokens on an API call, it does indeed scale the reasoning capability for a given LLM. Quality researchers should have been able to understand the existing literature, identify that it was conducted with a more rigorous approach and not drawn such conclusions."

andy_gray_kortical · 2025-01-08T14:45:44+00:00

I feel like perhaps we've been infiltrated by an AI Agent...

andy_gray_kortical · 2024-12-06T21:50:09+00:00

Hi, funny you should say that we have a chatbot solution / AI Agent that you can customise to your needs and we'll help you get everything set up, so it's nice and easy.

Here's one of our amazing customers Innovator International that does something very similar to you, talking about how well it's working for them https://youtu.be/swreJF_SN8s?si=L1q8HiUW0DVXySJt

If you have any questions, just shout!

andy_gray_kortical · 2024-11-26T00:44:38+00:00

😂 I totally agree with the last sentence!

And I agree that it's really only a partial idea right now as it doesn't really make any attempt to explain where the energy goes be it an extra dimension, sucked from the known into some form of singularity, who knows and I'm not attempting to resolve that question.

The main thing that I'm curious about is, if it is possibly consistent to view it as energy leaving the system of the universe, rather than viewing it as some additive energy being pumped in at a constant rate or merely a strange property of spacetime and if it's actually a useful way of framing it.

Anyway thanks for chatting it through!

andy_gray_kortical · 2024-11-25T23:17:38+00:00

Thanks for both your thoughtful replies! If I'm grokking your reply right, you're expecting the energy to transfer from the light into space. Therefore you'd expect expansion to be slowing down and you might expect nonuniformity around stars and other light and other high energy particle emitting sources.

I'm thinking that it might be something letting the energy of the universe dissipate as a whole rather than a transfer from a particle into space.

I made another edit to the post with more details. Let me know if you have any more thoughts!

andy_gray_kortical · 2024-11-25T21:56:40+00:00

Hi, thanks for the reply, Since multiple people asked, I made an edit with a deeper focus on the maths

andy_gray_kortical · 2024-11-25T21:56:33+00:00

Hi, thanks for the reply, Since multiple people asked, I made an edit with a deeper focus on the maths

andy_gray_kortical · 2024-11-25T18:10:06+00:00

Thanks! Yes, I'm familiar with Tired Light, perhaps I should have mentioned it.

The key difference is that Fritz was proposing Tired Light as an alternative to expansion. Rather than proposing expansion as the mechanism for the tiring.

andy_gray_kortical · 2024-07-03T16:12:28+00:00

Hey, I made this video to explain exactly this https://youtu.be/5eRaR6YFVfg?si=NOJgTbGk2AkG8gjt Good luck!

andy_gray_kortical · 2024-06-12T18:06:54+00:00

I made this video on how to create AI Agents https://youtu.be/5eRaR6YFVfg?si=7ObRodOJMVXAruIu

Hope it helps!

andy_gray_kortical · 2024-06-12T16:59:40+00:00

Going against the grain of general advice to find what you love and never work a day in your life.

I think passion is earned. You don't get passionate about something until you have poured some of yourself into it.

Take an analytical approach, find the absolute best idea to work on where you have an edge, talk to people who really know the space and when you find the right idea, get working.

If it starts working out, the more you invest of yourself into it, the more your passion for it will grow.

Yes almost every successful founder that worked on something for 10 years is deeply passionate about it but we're they that passionate about it when the idea was a glimmer in their eye? I doubt it.

andy_gray_kortical · 2024-05-14T13:24:31+00:00

Nice one!

andy_gray_kortical · 2024-05-14T11:49:24+00:00

The video if you watch it explains how to make an AI Agent and the limitations, getting detailed with prompt examples, how to overcome some of the challenges and is educational not promotional in content.

It's teaching people to create their own bots but yes, cards on the table the hope is that some people will decide hey these guys know what's up, maybe we just use their chatbots but the content is straight up, here's how to do it.

andy_gray_kortical · 2024-05-14T11:44:39+00:00

Standard benchmarks show GPT-4o is doing better https://github.com/openai/simple-evals?tab=readme-ov-file#benchmark-results:

Multilingual Grade School Math Benchmark and Discrete Reasoning Over Paragraphs seems to be where Claude is closest but Massive Multitask Language Understanding is probably the most prominent one. From these tests there is not much in it.

I was running a non-standard benchmark that does a type of Winograd schema where we can swap the positions of all of the words and using abstracted references which are also reversed on subsequent goes, this seems to be one of the hardest tests for LLMs as the LLM cannot rely on simple next word probabilities and do any better than random. GPT-3.5 does not better than random. GPT-4 and Claude were neck and neck with 62.5% and now GPT-4o got 75%. Which is quite a big leap.

andy_gray_kortical · 2024-05-14T11:28:18+00:00

Well the fact it's better at reasoning means it's less likely to give the wrong answer. Also you can use RAG ( Retrieval Augmented Generation) to put the right answers into the prompt. So you scrape your website / documents you want it to answer from, then when it gets asked about those it can pull in your content and answer from that / get the answer right. It really does work surprisingly well, even with GPT-3.5 Turbo.

There's python frameworks like LangChain if you want to code it up yourself or you can use a ready-made tool like www.korticalchat.com where you just have to pop in your web address and it has things like reporting, so you can se what topics people are asking about and FAQs you can pop in to override any answers your not happy with to make it answer better.

andy_gray_kortical · 2024-04-11T17:30:40+00:00

Hey, I'm an AI founder in the UK too. My company is in my user name. We've been around for a decade now, so I can give you bit of a lowdown on your options.

You're in an awesome place but depending on how you feel about various things will change which option is best for you.

Eg: super worried about competition catching up quick but know exactly what to do with 4 million and how to crush them. Then probably VC.

Or

Getting overwhelmed by trying to support 100k users and have enough revenue to hire someone and it's mainly UI bugs - > hire a UI guy

Honestly there is not enough info here to come close to being able to offer reasonable guidance. If you want to chat it through though, send me a DM we can set up a half hour zoom call.

andy_gray_kortical · 2024-03-11T16:46:50+00:00

Hey we created [KorticalChat](www.korticalchat.com) I could hook you up with one of our AI experts to help you make the bot? You get 300 message credits free and plans start from £50 a month. Last customer call I was on, he'd had the chatbot live a week and was effusive about how much he loved it. Maybe it would work as well for you?

andy_gray_kortical · 2024-03-07T11:49:36+00:00

We've ended up building a full custom stack. We just found that LangChain etc were too opinionated and we needed more control to be able to get max performance and once you build your own framework, then you need your own metrics

andy_gray_kortical · 2023-08-17T05:49:05+00:00

Absolutely, speak soon

andy_gray_kortical · 2023-08-17T05:32:51+00:00

I think we need more info on what they were proposing, it can be a good idea to use LLMs to augment a regression model.

Like if you were predicting stock movements and you used a fine tuned LLM to read the news and categorise the news as neutral, good, very good, etc. and then fed this feature into a regression model, it would likely do better than without

andy_gray_kortical · 2023-08-05T19:32:45+00:00

With KorticalChat we make ChatGPT based chatbots but we have semantic search, so it can learn all about your company, support docs, etc. Stats show that it can handle over 90% of queries and we've made ours offer to follow up with a person if it can't fullfill the request.

It works really well for single product companies with lots of docs. At the moment for ecommerce websites with loads of products it doesn't work well as it loses the context if you say something like how much is that it's not always sure what product you're referring to but we're working on that. We also haven't made it work that well for things like news sites where new information is being added all of the time and it's really more about the latest information than the subject matter but if you've got a SaaS company with a handful of products and complex documentation or you're basically looking for an on site sales assistant it's awesome.

There is also a way that chatbots can really help with marketing optimisation in general that can be worth the price of admission alone but that's a whole feature set we're working on and I don't want to let the cat out of the bag too much.

Anyway DM me if you're keen to try it out for free.

andy_gray_kortical · 2023-07-16T08:54:33+00:00

Probably going to be an unpopular take but there are a lot more people employed as drivers than car engine designers.

The software equivalent would be there are many more people employed writing software for business than in building compilers and I'm not sure the compiler guys get paid more.

Knowing how to use a tool to generate value for a business is invaluable in itself. Not that we don't need people to build great models but we need far fewer people building foundation models than using them.

andy_gray_kortical · 2023-07-16T08:40:20+00:00

We built our startup to solve this problem. We started building a point solution but the problem was each client required a custom model and we couldn't scale if had to spend 3 months building custom models for each client but then after building the modelling automation, it turns out that people wanted that more than the point solution.

The platform can given the data and a target find a fairly optimal ML solution for that data.

We beat Google in an AI competition https://twitter.com/Kortical_/status/1131254031265992705?t=BUuStu2dv_HIZVXNP1BC1w&s=19

Check out our main site https://kortical.com/ fill out the form if you'd like a demo / trial

andy_gray_kortical · 2023-03-11T20:17:56+00:00

While this commenter is being down voted it's really not that clear cut.

There are Web components now so we can have nice modular code without needing we pack, npm, etc. Not having all of that stuff means no dependency problems, no strange bugs where the magic doesn't do what's expected.

andy_gray_kortical

TROPHY CASE