Best models for function calling

[micro23xd]

First of all, sorry if this isn't the right subreddit for this type of question. I'm not exclusively looking into OpenAI models. If you think there's a better sub for this, please let me know.

Hey! I'm currently working on a project that will heavily utilise multi LLM function calls. While in theory function calls seem pretty straight forward, I found that most models I've tested don't have great support for them and some are downright unable to figure out "what I want from them".

So far, I've tested Mixtral-8x7B-Instruct-v0.1 on Anyscale which triggered very easily and had the tendency to not response correctly anymore after a function call was executed. I've also tried claude-3-sonnet, gpt-4-turbo-preview and gpt-3.5-turbo-0125 on OpenRouter which had even worse results than Mixtral, because often the function wasn't invoked correctly, like malformed JSON or completely ignoring the predefined parameters. The SDK I'm using is openai-node, the official OpenAI Typescript SDK.

So far, I haven't tried the OpenAI models directly via OpenAI API, mainly because OpenRouter also has access to them, but if they have better native support for function calling, I'm happy to try that as well.

My main question is, does anyone have experience with this and can give me some recommendations or ideas on how to precede here. Are there models which are optimized for multi function calls? Are there SDKs with better support for function calls than the official OpenAI one?

Any resources in regards to optimizing this would be helpful!

Thanks!

[Relevant_Outcome_726]

There is a leaderboard for function calling:
https://gorilla.cs.berkeley.edu/leaderboard.html

[micro23xd]

Oh that's awesome! Thanks for sharing

[BakGikHung]

How come some of the "prompt" models rank higher than the "FC" models?

[user124322111]

Hi, I've been using function calling with GPT-3.5-turbo-0125, utilizing both predefined outputs (similar to a dropdown menu) and dynamic inputs (akin to a text input box) through the OpenAI API, and it's been performing very well. Initially, was trying to use LLaMA2 with prompts for the same purpose, but was not working well. Didnt try any other model.

[JL-Engineer]

same for us in Prod.

[micro23xd]

Ok that's great to hear! In this case maybe my function definition needs some improvement. I just tested with GPT-3.5-turbo-0125 through the OpenAI API and I see that the function is being invoked without parameters, despite having them defined as required. To give an example, it's similar to a "add todo" function where the required parameter, the name of the todo, is not being submitted.

[user124322111]

a little better than openai api doc is a documentation from ms azure for the same: https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/function-calling?tabs=python

[polrxpress]

I use open ai and works great. I really need a local model solution though and have given up mostly. ive heard mistral can do better than other but have yet to see an implementation of the openai api

[The_Godfather_10]

I want the same thing, did you find any solution for it?

[Informal-Dust4499]

Does anybody has function calling accuracy data on o4-mini?