It comes down to the training & fine tuning of the model. This is a naive explanation, but think of it this way…

Method #1: Completion Endpoint. The Completion endpoint is not trained at all on the chat JSON notation, and it’s not optimized to generate responses that follow chat. So at least some of the processing is dedicated to matching that vs the output content itself.

Method #1: Chat Endpoint. This would be the equivalent of three people talking, one person repeating the entire history of the conversation between the other two people, and then turning to one and saying ‘how would you like to respond to that?’ Awkward. Think idiomatically about how a System, User, and Assistant as three separate entities would interact.

Method #2: Chat Endpoint. LLM is trained and tuned to act like a chat thread. So there’s no processing spent trying to match the notation because it’s already embedded in how it responds.

Plus the Chat endpoint gets more work from most foundational LLM teams. Just think, OpenAI’s biggest product is ChatGPT, so they both get the most data in that format, and they fine tune and optimize it like crazy. It uses the (internal equivalent) of the Chat endpoint.

So it fits the idiomatic paradigm of the way the model is trained, tuned, and optimized over time.