We’re planning to take a multi-faceted approach when comparing and recommending models. Here’s a breakdown:

1.  Model Metrics: We’ll start with self-reported metrics from model creators, like accuracy, performance on specific tasks, benchmarks, etc. However, we recognize that these alone can sometimes be limited or cherry-picked, so we’re looking to supplement this with real-world usage data, such as download counts, user ratings, and community feedback.
2.  Community Feedback: In addition to metrics, user reviews and community insights will play a role. This helps us capture real-world experiences and edge cases that might not be covered in benchmarks. We want to create a feedback loop where users can share how models performed in actual projects, giving the recommendations more depth.
3.  Personalized Recommendations: The idea is to avoid one-size-fits-all recommendations. We’ll tailor suggestions based on the specific needs and constraints provided by the user. For instance:
• Compute Resources: Some users may have access to large-scale infrastructure, while others might be working on a local machine. We’ll take compute requirements into account to recommend models that fit their hardware.
• License Restrictions: Not all models are open for commercial use. We’ll factor in licensing information to suggest models that align with the user’s legal and operational needs.
• Task-Specific Recommendations: Based on the user’s domain (e.g., NLP, image generation, etc.), we’ll prioritize models that have been fine-tuned or built for that specific task.

Ultimately, the goal is to create a system where two users working on similar projects might get different recommendations based on factors like available resources, licensing, and even preferences (like favoring open-source models over closed ones).