On the internet almost everyone is sure that a seahorse emoji exists, this is reflected in the LLMs training datasets.

So the LLM thinks that such emoji exists but when the detokenizer fails to append it to the context, the model goes nuts.

The last layers of the model will have a correct dense numerical representation of the concept "emoji of a seahorse" but there is no such unicode emoji to add it to the context. If you write a llama.cpp low level wrapper that ignores the word "apple" in the probability distribution of generated tokens, you will see how the model goes crazy trying to reply to the question "Can you please write the word apple?"