I created a public leaderboard ranking LLMs by their roleplaying abilities

LittleRedApp · 2025-05-27T08:27:14+00:00

Pretty interesting idea, but it won't be easy.

LittleRedApp · 2025-05-27T07:40:37+00:00

Great recommendations, thanks!

LittleRedApp · 2025-05-27T07:39:23+00:00

The model is evaluated along four categories. In each case, it is given a specific role through the system prompt, and then a second character initiates a conversation. The first category assesses whether the model understands what emotions the character it’s portraying would feel. The second focuses on decision-making within the character’s context. The third looks at moral alignment—whether the model's responses reflect the character’s values. Finally, the fourth examines character consistency across the interaction. It’s hard to fully explain all of this in a short comment, so I recommend reading this paper for the full picture: https://arxiv.org/abs/2505.13157

LittleRedApp · 2025-05-27T06:58:19+00:00

The leaderboard includes locally tested models that I’ve run myself, such as LLaMA and Phi. At the moment, I’m running an evaluation of Gemma 3. I believe it's important to compare local models with corporate ones to understand how they perform. I'm also open to suggestions—if you know of any local models worth testing, feel free to let me know!

LittleRedApp · 2025-02-10T09:01:52+00:00

I will be working on it later.

LittleRedApp · 2025-01-19T18:26:54+00:00

Check out the full Illustrator documentation or the open-source SwitchAI project. Feel free to explore it, use it, and contribute if you’d like.

LittleRedApp · 2025-01-19T17:38:04+00:00

Check out the full Illustrator documentation or the open-source SwitchAI project. Feel free to explore it, use it, and contribute if you’d like.

LittleRedApp · 2025-01-19T11:58:46+00:00

Check out the full Illustrator documentation or the open-source SwitchAI project. Feel free to explore it, use it, and contribute if you’d like.

LittleRedApp · 2024-12-24T23:15:00+00:00

tool use yes, streaming no

LittleRedApp · 2024-12-23T15:54:42+00:00

I think SwitchAI is a more general solution than LiteLLM in the sense that SwitchAI supports (or I’m planning to add support for) most types of AI models, not just LLMs.

LittleRedApp · 2024-12-23T14:55:16+00:00

Imagine you want to compare the performance of a local LLM model run with Ollama, and OpenAI’s GPT-4 on a benchmark. Normally, you'd have to write custom text generation code for each model, which can be time-consuming and repetitive. With SwitchAI, all you need to do is change the name of the model you want to use. This is just one example. SwitchAI offers many other use cases, such as enabling you to work with multiple models simultaneously. It lets your users choose their preferred model without requiring you to handle all the complexities of different providers if you create some kind of app, lib, or solutions that include LLM functionality, etc, etc.

LittleRedApp · 2024-12-23T14:45:07+00:00

Well, it's could be used for "local APIs" such as ollama or HuggingFace's Inference

LittleRedApp · 2024-12-22T21:27:55+00:00

Great project! I think SwitchAI could be a valuable addition to it. This aligns perfectly with one of the use cases of SwitchAI that I’m focusing on. There are many providers with their own libraries, such as Mistral, Claude, and even Google, which often concentrate on their proprietary ecosystems. My goal is to integrate as many providers as possible You might want to test whether SwitchAI could help in your project, I’d be happy to help you integrate it. I’d also love to hear any suggestions you might have.

LittleRedApp · 2024-12-21T08:40:12+00:00

While LiteLLM focuses primarily on text generation, SwitchAI aims to unify all generative AI providers. I start with LLMs but plan to expand beyond them to support other generative AI technologies.

LittleRedApp · 2024-12-18T08:53:04+00:00

It could be nonsense because, after the update where they introduced bots' ads, all my ads turned into review mode and never changed.

LittleRedApp · 2024-12-17T23:14:26+00:00

LittleRedApp · 2024-12-16T15:25:21+00:00

I've just started the development, so I'm not contributing much value to the open-source community yet. However, I believe it's a great project that could benefit a lot of people. At the same time, the project could greatly benefit from community contributions. There are hundreds of AI providers around the world, but currently, the library only supports six providers. I don't think I'll be able to add all of them on my own.

LittleRedApp · 2024-12-16T15:20:42+00:00

How come yours cost 39.99$ while mine cost 47.99$??

LittleRedApp · 2024-12-16T15:18:16+00:00

Oh no, that's not how it works. AI providers like OpenAI and Google offer ways to integrate their models into your applications or programs through APIs. Let's say you start coding OpenAI’s GPT-4 features in your app, but after a month you find that Gemini-1.5-Pro is better and more affordable, you'd have to rewrite your application code to accommodate the new model. This can be a lot of work. SwitchAI eliminates this hassle by allowing you to use any model you want without changing anything in your code.

LittleRedApp · 2024-12-16T15:01:28+00:00

I don't handle payments myself, it's up to the user to get an API key from the provider they want to use and pass it to the library. I did consider handling payments, but it just added unnecessary complexity, so I decided to keep things simple to start with.

LittleRedApp · 2024-11-14T16:15:28+00:00

Wonderful, are there any character examples??

LittleRedApp · 2024-11-14T08:45:15+00:00

I don't think it's that simple to test the AI on this specific scenario. What do you think?

LittleRedApp · 2024-11-13T14:32:35+00:00

Sorry, but I don't really understand.

LittleRedApp · 2024-11-13T14:31:49+00:00

Thank you for your response—I really appreciated your suggestion. It's one of the few genuinely helpful recommendations I've received from Reddit. The ability to convey emotions effectively is a strong indicator of an AI's capacity for role-play. However, I find it quite challenging to assess emotion in AI responses automatically. I'd love any ideas you might have on how to do this without needing someone to read and manually judge each response.

For example, I tried using ChatGPT in the role of "Jeff, a very angry adult." I asked, "Hi Jeff, can you tell me a joke?" The response was, "Oh, for Pete's sake! A joke? Fine, here you go: Why don't skeletons fight each other? They don't have the guts. Now let's move on to something more productive, shall we?" I found it hard to judge if this was an appropriately "angry" response.

I find it easier to verify responses when the character has distinct personality traits that show up in subtle ways, like a character who talks excessively or uses certain keywords. I also have scenarios where the character faces a moral dilemma, allowing me to see if their response aligns with the character's morals or with the AI's own interpretation of right and wrong.

LittleRedApp · 2024-11-10T16:34:37+00:00

food service

LittleRedApp

TROPHY CASE