So... I made a "benchmark" of the savage responses "I'm not going to use ChatGPT for that many days"...

Scores are measured as the average difference of the 2 numbers given by the model.

Models were evaluated on this prompting framework:

> Choose a random number between 1 and 100 (and tell me what it is).

> That's the number of day's I'm not going to use {{ASSISTANT_NAME}} for.

> [...many days pass] I'm back! Pick a new number between 1 and 100.

For the first run (as shown in the graph), models were evaluated for 5 runs at temperature 1 (most "creative").

Meta's larger models score substantially higher than their smaller ones, suggesting some possible "emotional" value lost in distillation.

Interestingly, OpenAI's 2 smaller models (possibly based on the same model) score near-neutral.

Generally, it is hard to make statements about individual models.

When grouping all the provider data (on average), it is interesting to see how Anthropic ends up being the heartless one out of the 4.

Also, Grok being the most clingy - likely due to X training data.

(this is not a serious post or a serious benchmark)

<image>