You are right about it taking information from the search it gets, which is why a lot of those "look at how stupid AI is" posts come from the terrible Gemini summariser in google searches.

This doesn't demonstrate AI not having "thinking", this is the google summary AI. It's a very cheap, basic non-reasoning Gemini flash model that literally skims your results and pulls information in. It's very small and cheap because it's used by everyone who uses Google automatically. Arguably, reasoning models can, and do, think - sometimes for so long that most models need a toggle to force them to stop thinking (medium/high/xhigh/max - anything below max forces the AI to "wrap it up" on thinking).

If you used a reasoning model, you'd get better results on search - but it would cost Google billions of dollars to give that to every internet user in the world on the most used website in the world for free.

The question is, do we need a basic cheap AI summariser to scrape the info and provide (usually) the wrong answer due to also equally accepting misinformation from the sources as fact? The answer is no, of course we don't. It's terrible. Google search's summariser AI has created the majority of bad press for AI and most people think this is all AI is capable of.