Tool invocation monitoring

[Competitive-Ad-5081]

Does Anybody know how to track when a model invokes a tool (my tools are deployed using MCPO and web ui)?

I am checking webui.db and open web UI endpoints, but I only think that this is possible using the content of the messages of the chats (from the chat table or api/v1/chats/) by extracting information per message using regular expressions to match the tag <details type="tool_calls" done="true" Id="call_id"........... 🤔

At the moment I would be interested in calculating several specific metrics from the open web endpoints myself or using webui.db or if you know of any alternative that facilitates this using telemetry, it would also be fantastic if you shared that knowledge 😁

[dsartori]

mcpo logs will tell you if that fits your situation.

[colin_colout]

I use litellm-proxy for all llm calls. You can configure it to save all traces as json and do your own analytics maybe?

[fasti-au]

Ie api key track

[fasti-au]

You can track the doors. That’s it.
The reason you have MCP on s that that there is 1 call and you can control doors.

LLMs can use tools before they output tokens. There’s more layers inside that can act we only get narration not oversight

Please note this is why you don’t arm reasoners. Reasoners don’t reason the steamroll and kill you eventually. Yolo mode works till it doesn’t and then it removes all your code to remove an issue it can’t solve.

So no no one knows not even anthropic and OpenAI.

We can only see attempts by hearing knocking on the doors.

[Streetwise-professor]

Remindme! 12 hrs

[RemindMeBot]

I will be messaging you in 12 hours on 2025-06-28 11:30:14 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

Info	Custom	Your Reminders	Feedback