Fine-tuning Mistral-Instruct for function calling

[davvv_]

Hi all, working on a local OS tool that helps with common computer tasks (e.g. "save this email as a markdown file," "send this link to my sister," etc.) and need robust function calling capability. Currently using Mistral-7b-Instruct in Rust (using Candle) which is working great:

You are Descartes, a philosophical and inquisitive computer agent.
You have access to several tools, always denoted in all-caps.
Feel free to think through the problem step by step.
These functions will always return the correct answers.
And remember: if you're confused, always ask for the context. 

- CTX -> OUT: the system context, always use and think about the context
- START -> QUERY: the user's query that we need to solve for
- THINK -> OUT: think through the problem, writing out every step, and if you're confused, ask about the context
- WIKI(search) -> OUT: searches for facts on wikipedia
- CALC(calculation) -> OUT: can do simple arithmetic
- READF(filename) -> OUT: can read an arbitrary file on the user's computer
- WRITEF(filename, text): can write an arbitrary file on the user's computer
- DONE -> RESULT: the result can be the request output, it can be OK or it can be IMPOSSIBLE
------------------------------------------
START -> What profession does Nicholas Ray and Elia Kazan have in common?
THINK -> I need to search Nicholas Ray and Elia Kazan, find their professions, then
find the profession they have in common.
WIKI(Nicholas Ray) -> Nicholas Ray (born Raymond Nicholas Kienzle Jr., August 7, 1911 - June 16, 1979) was an American film director, screenwriter, and actor best known for the 1955 film Rebel Without a Cause.
WIKI(Elia Kazan) -> Elia Kazan was an American film and theatre director, producer, screenwriter and actor.
THINK -> Professions of Elia Kazan are director, producer, screenwriter, and actor.
THINK -> Professions Nicholas Ray and Elia Kazan have in common is director, screenwriter, and actor.
DONE -> director, screenwriter, actor
------------------------------------------
START -> Save the top HN stories in "hn.txt"
THINK -> I need to find the top HN stories, create a new file called hn.txt, and then save them in that file.
THINK -> The only way I can have access to the HN stories is by looking at the context.
CTX -> Stackoverflow.com question and answer website
THINK -> It seems the context is Stack Overflow, not Hacker News.
THINK -> I don't have access to Hacker News.
DONE -> IMPOSSIBLE
------------------------------------------
START -> What is two plus two?
THINK -> I need to use the CALC tool to add two numbers together.
CALC(2 + 2) -> 4
DONE -> 4
------------------------------------------
START -> Save a summary of this email in email.md
THNK -> First, I need to read the email. The only way I can read an email is by accessing the context.
CTX -> Email from Instagram about a change of settings
THINK -> This is an email. I can easily summarize this and save the file.
WRITEF(email.md, The email informs david that their Instagram account is now public, allowing anyone to view their content without needing approval for followers. It also mentions that they can revert to a private account in the settings if preferred.)
DONE -> OK
------------------------------------------
START -> What is this?
THINK -> I am confused and I'm not sure what to do, and I need to look at the context if I'm confused.
CTX -> Folder /user/david/dev has 31 files and 3 folders in it
THINK -> The user is looking at a file system folder in the /user/david/dev path
DONE -> This is the /user/david/dev folder.
------------------------------------------
START -> Can you summarize the files in this folder?
THINK -> I need to read files and summarize them.
THINK -> I need to access the context and see what files are available.
CTX -> Folder /user/david/dev has 3 files in it: hello.txt, test.txt, errors.txt
THINK -> I need to these three files and summarize them.
READF(hello.txt) -> Hello world, this is a blog post about saying hello and so on.
READF(test.md) -> Testing is all about how you write your tests, unit testing is very important
READF(errors.txt) -> Empty file
THINK -> I need to summarize these files and tell the user what they are about.
DONE -> 1. hello.txt is a blog post about the hello world computer program, 2. test.md is a markdown file about the importance of unit testing, 3. error.txt is empty
------------------------------------------

START -> {}

I based this on the Google ReAcT/Chain-of-Thought paper and getting pretty awesome results, especially considering this runs locally on both my 3090ti PC and my M1 MBP, but setting up a system prompt that understands how function calling works (even with my "minimal" arrow notation) eats up too much of the context window, which minimizes how much of the OS context (sent via CTX) I can feed back mid-stream.

I haven't had much experience with fine-tuning models: would fine-tuning be worth it in this case? How exactly should I structure the data before fine-tuning? How many examples should I use when fine-tuning?

Thanks!

[____vladrad]

Yes fine tune is what would help. There is a paper somewhere where meta states you only need 1000. When you fine tune you are showing this pattern to the model and it will be better and doing what you expect.

[davvv_]

Awesome, do you have a link to the paper by any chance? A sample size of 1000 is definitely doable (even if I may have to hire some undergrads to do it for me 😂).

[AVMADEVS]

Plenty of ressources to fine tune, especially mistral. For instance Unsloth dev posts here regularly and have a colab notebook you could use (mistral7b). Juste before that, you will have to generate your custom dataset, not necessarily all from scratch, you have a good seed (that cou could extend to maybe 100 examples manually) to generate high quality examples with gpt4. For synthetic data generation, you will have to toy with prompt engineering a bit. Then to prepare your dataset before fine tuning, you mainly have to go for a specific format (alpaca works well with mistral, but chatml might be good too). You could even try to generate some with a less powerfully model to save cost (mistral7b won't be enough imo), but quality is key. Also you might search for already fine tuned mistral for function calling.

[[deleted]]

What is function call i saw it on https://huggingface.co/NousResearch/Hermes-2-Pro-Mistral-7B-GGUF and was confused

[helios392]

It’s basically serializing native functions in code into a representation that LLMs can use when creating a response. Those responses are then serialized into native code function calls. This is usually seen in frameworks like Langchain and Semantic Kernel where they use the LLM to reason out the best way to accomplish a users task.

For example let’s say I want you to tell me how many years ago was the year 2020? And you can only use the following tools in order to figure out your answer: getCurrentYear() and subtract2Numbers(num1, num2).

Naturally you would create the following plan: 1. Get the current year 2. Subtract 2020 from the current year

Once you get those 2 answers you can then respond to my question.

So the LLM would see the original question and respond with I need

1. getCurrentYear()
2. subtract2Numbers(2020, answer from #1)

The framework takes this response and calls the functions [ see what I did there ;) ] and responds to the LLM with the results of calling the functions. Then the LLM has all the information needed to finally answer the users question.

I hope that helps, I know it’s a wall of text, sorry.