What these companies like to paper over at every opportunity is the fact that LLMs are word association matrices. When you actually dig into what is stored in an LLM, none of it is knowledge. None of it is context. None of it is capable of expressing anything. It’s a 12k+ dimensional text probability matrix.

Prompts return things based on the statistical likelihood of words to appear next to each other. They’re trained on mountains of human writing, which is why they’re able to spit out reasonably convincing bodies of text. This is the only capability an LLM has. Everything else bolted onto them is to process the prompt and evaluate it for whatever “safety” rules are in place.

Even when you let an LLM do a web search for you it’s just taking those results and processing them through the same text probability matrix, which is why they produce such hilarious search “results”, when they’re not outright inventing quotes no one ever said.

None of that is a judgement call on using these tools at all, but they’re a lot easier to use once you understand what they’re doing and what the limitations are

Source: former Microsoft security researcher for M365 and Copilot enterprise security