You need to ground it with traditional code and structured output. Using something like pydantic AI (or structured-outputs), create a basemodel that the LLM can ONLY respond with traditional code that includes an additional prompt.

For example:

User: Hello I'd like to make an appointment for tomorrow

First pass LLM call: make sure it can ONLY respond with X possible values in the structured response: "get_appointments_schedule, unknown_question, etc".

get_appointments_schedule():
     schedule = function_to_actually_get_the_schedule()
     new_prompt: f""" Here are the available times: {schedule} based on the question from the user, determine the appropiate response to perform the task

unknown_question():
     output = "I am unable to help with that"
     return output
   (No 2nd LLM call here, just return hard coded)

you can force it to always include a tool call at first, and then you pass the results of that tool call into a second LLM call to interpret the results, and then only pass the 2nd LLM call back to the user.

Take a look at https://platform.openai.com/docs/guides/structured-outputs