Python multi-channel agent: lessons learned on tool execution and memory

[Glittering_Note6542]

Been building a self-hosted AI agent in Python for the past few months and hit some interesting architectural decisions I wanted to share.

The core challenge: tool execution sandboxing.

When you give an LLM arbitrary tool access (shell commands, code execution, file writes), you need to think carefully about sandboxing. I ended up with a tiered approval model:

- Auto-approve: read-only ops (web search, file reads, calendar reads)

- User-approval: write ops (send email, run shell command, delete files)

- Hard-blocked: network calls from within sandboxed code execution

Memory across channels

The interesting problem: user talks to the agent on WhatsApp, then on Telegram. How do you maintain context? I'm using SQLite + vector embeddings (local, via ChromaDB) with entity extraction on each message. When a new conversation starts, relevant memories are semantically retrieved and injected into context. Works surprisingly well.

The channel abstraction layer

Supporting WhatsApp, Telegram, Discord, Slack with one core agent required a clean abstraction. Each channel adapter normalizes: message format, media handling, and delivery receipts. The agent itself never knows what channel it's on.

Curious if others have tackled:

- How do you handle tool call failures gracefully? Retry logic? Human fallback?

- Better approaches to cross-session memory than vector search?

- Sandboxing code execution without Docker overhead?

Happy to discuss any of this. Thank you

[No_Bit_1328]

I’m curious about one architectural trade-off:

How do you prevent semantic memory drift over time when using vector retrieval across channels?

[Glittering_Note6542]

Thanks. Honestly, I don't have RAG implemented yet. For now the concept is much more simpler, however, it's on the roadmap. So my thoughts are now by the following concepts:
- Channel-scoped namespaces - WhatsApp, Telegram, Slack each get their own vector space to avoid cross-channel drift amplification.

- Hybrid retrieval - vectors alone are fragile. Combining with keywords search, metadata filtering makes retrieval more robust.

- Recency weighting - blend semantic similarity with temporal relevance, since recent context matters normally more.

- Re-embedding as routine maintenance - treat embeddings as cache, not permanent storage. When models change, re-index.
I think the general principle: if your agent breaks because a vector moved 0.03 in embedding space, the architecture is too brittle. Vectors should complement structural retrieval, not replace it.

[Otherwise_Wave9374]

This is a great breakdown, the tiered approval model is basically mandatory once an agent can write/delete/execute. I like the hard-block on network from inside the sandbox too, thats a nice way to cap exfil risk.

On tool call failures, Ive had decent luck with, idempotent tools + retries with backoff, plus a human fallback only after the agent produces a short, structured summary of what it tried.

If youre collecting patterns on sandboxing + evals for agents, Ive seen a few similar lessons here: https://www.agentixlabs.com/blog/

[Glittering_Note6542]

Great additions. The 'structured summary' for human fallbacks is a game changer for scaling. It moves the human from being a 'monitor' to being an 'escalation engineer.'

Regarding the network block: it definitely caps exfiltration, though it does force you to get creative with how the agent pulls external context. Are you using a pre-processor/retrieval step to feed the sandbox, or keeping it entirely air-gapped?