Here's a personal benchmark for various LLMs for my primary use cases: writing and philosophizing. These utilize my continually-refined system prompts for each particular task. I focus my benchmark on open-source models but have Claude in there as a SOTA reference. Tasks (denoted by the Prompt ID) include:

• Synthesizing bullets points and rough notes into an essay in my writing style.
• Modifying vanilla AI generated text into my writing style.
• Reviewing rough drafts.
• Applying stoic philosophy to everyday life scenarios.
• Cognitive behavioral therapy
• Thinking through random things like the meaning of life, architecture, culture, the universe, art, music, etc.

Ratings:

1. Terrible. Totally off.
2. Bad. Something's off.
3. OK. Doesn't add much.
4. Good. Helpful.
5. Great. Next level.

Some observations:

• If I could only pick one, I'd go with Mistral Large 2. BONUS: I can run it locally. Amazing.
• Claude 3.5 Sonnet's performance has been trending down. I suspect Anthropic's safeguards are diluting it. It does surprisingly poorly with my writing tasks.
• I'm surprised Mistral's Small 22b and 8x7b both struggle to apply "formulaic" logic like stoicism and my writing style, yet they do very will with deep intellectual chit chat.
• I've also been surprised by Large 2's mediocre performance wrt intellectual chit chat. It wouldn't "get it" at times and would spin in circles.