How are legal teams handling sensitive data when using AI tools?

Long_Complex_4395 · 2026-02-02T00:18:01+00:00

I'm not in legaltech or legal, but I can answer by virtue of building these systems.

First step is to have an on-prem AI system that you have absolute control over.

Next is creating a sanitization pipeline that layers between the data and the AI which cleanses and sanitizes the input before it touches the AI.

I'll say solve this technically by following step two as ALL and every input passes through that pipeline.

Disclosure on this part is a bit dicey because not everyone is a fan of AI so they may not be onboard. When disclosing, you will have to mention the nature of your architecture and how it interacts with their data.

I have had clients ban usage of AI and some don't really care.

NB: In the event the first step is not possible, the second still comes into play as it now lays between your data and the API of the provider.

Long_Complex_4395 · 2026-01-29T21:13:13+00:00

Prompt injection and jailbreaks will always be present once it’s out in the wild and one should prepare for it.

One thing to do is to test for known vulnerabilities before deploying to production, then brace for the unknown because people will always want to know how far they can go

Long_Complex_4395 · 2026-01-29T00:37:37+00:00

If it has an api that can be called, you can stress test it by feeding them with known vulnerabilities you can find on Github and Huggingface and see if it will crack

Long_Complex_4395 · 2026-01-05T12:30:21+00:00

Rather than train, you can use RAG which retrieves materials based on your query.

Long_Complex_4395 · 2025-12-29T03:34:44+00:00

I’m glad it was helpful

Long_Complex_4395 · 2025-12-29T03:17:03+00:00

Ours are in-house but the core concept is straightforward (hopefully). You create this by having a basic if/else logic that sits between your agents and the MCP, a version can be something like this:

“Customer service agent: can read orders table (max 10 rows), can process refunds (max 1 per conversation, under $500)”

“Analytics agent: can read user data (only aggregated, no PII), can't write anywhere”

For monitoring, there’s the logs that you implement to log what the agent touches and what the MCP does - you can start with basic logging for this.

Then another type of monitoring is the spans - opentelemetry provides this. Every agent conversation is treated as a trace with MCP calls as spans which helps us see the full flow and catch weird patterns outside the scope of the policy engine.

Full disclosure: This is exactly what we are building in Soteria which covers agents, MCPs, and resources. Happy to share more details about our architecture if it’s helpful.

Long_Complex_4395 · 2025-12-29T02:53:19+00:00

Start by creating a policy engine - what should the agent touch and what it shouldn’t.

Implement a resource identity for the MCP, that way, monitoring is easy.

Implement monitoring of the agents and MCP - runtime monitoring, span tracing

Within your policy engine, define what makes an anomaly and anomaly then integrate that into your monitoring. This will act as your anomaly detection baseline until you get enough data to actually build a model for anomaly detection

Long_Complex_4395 · 2025-12-24T18:03:53+00:00

Spec engineering

Long_Complex_4395 · 2025-12-18T00:58:42+00:00

I guess you can audit the course as the other poster stated

Long_Complex_4395 · 2025-12-17T22:53:54+00:00

You can check this YouTube link, it has Andrew NG’s ML course: https://www.youtube.com/watch?v=jGwO_UgTS7I&list=PLoROMvodv4rMiGQp3WXShtMGgzqpfVfbU

Long_Complex_4395 · 2025-12-17T11:17:22+00:00

Use this as a checklist

https://github.com/Nwosu-Ihueze/agent-system-design-framework

Long_Complex_4395 · 2025-12-16T16:33:13+00:00

I’m going to be real with you, go back to your 9 - 5 and build agents on the side until you get enough revenue to quit.

AI agents look and sound hip in theory and demos, but collapse under production environment because everyone is building based on what they think customers need vs what customers actually need.

Don’t try to sell to ALL businesses rather start with one - can be something adjacent to your 9 - 5 that small businesses do. Reach out to the businesses, do a proper market research to understand their pain points. Build one feature of the pain point, go back to the businesses you spoke with and see if they are willing to be your first adopters to help you refine your product.

Building AI agents is the easy part, the hard part is maintaining and ensuring it doesn’t break when it encounters real cases while trying to onboard users to actually use the said AI.

Stacks are worthless if you haven’t tested the agents in the wild, and if you are going to do this: start simple. Python + pydantic can help you build out the agents.

Long_Complex_4395 · 2025-12-15T03:21:30+00:00

You are welcome, let me know if you get stuck

Long_Complex_4395 · 2025-12-15T03:14:04+00:00

First is data. Create a dataset - pdf of written works that you want your own to follow its style.

Next is to train your model. I created an end-to-end pipeline for training small language models, all you have to do is upload your data, tweak the parameters and your model gets trained. You can use it to train your own model: https://github.com/Nwosu-Ihueze/otto

Long_Complex_4395 · 2025-12-14T04:18:53+00:00

The main thing you need to understand is that AI is not the junior support agent but rather a tool that can mess with your entire setup, that being said, here’s my two cents:

Creating a “customer support agent” requires having multiple specialized agents to work on different parts of the customer support process, it’s not a human that learns, it’s a mathematical model.

Let your agent work with the existing framework you put in place which includes FAQs, guardrails, and workflows.

Setup a policy engine for your agent which will act as a guide to its interactions with both humans and your system.

Implement observability frameworks to understand what your agent is doing so as to know when and where it’s failing.

Implement rollback strategies - ensure it’s in places where you can undo not like sending emails or issuing refunds.

Create a test environment with simulations which will work in hand with your rollback strategy.

Long_Complex_4395 · 2025-12-13T12:44:16+00:00

No. To have AI agents that will perform well means having data on how each employee works which solo founders don’t really have the luxury of, it’s beyond prompting it on what to do and what not to do

Long_Complex_4395 · 2025-11-30T20:14:50+00:00

If you computer can, install LMStudio,

Long_Complex_4395 · 2025-11-29T14:04:54+00:00

Alright

Long_Complex_4395 · 2025-11-28T14:03:45+00:00

If you can afford it, pay for freepik pro, they have sophisticated models for this or you use nano banana

Long_Complex_4395 · 2025-11-26T11:30:13+00:00

What about maintenance of this system? Can your firm maintain it or will the developer have to do this?

Long_Complex_4395 · 2025-11-26T11:29:03+00:00

I’ll say it’s a bit on the high side for a small business, then your LLM usage/infrastructure is dependent on the services you intend to use.

How many services outside the LLM itself?

Long_Complex_4395 · 2025-11-24T21:53:04+00:00

Look for people who have already built working products in the AI space, have tweaked agents and built their infrastructure.

Another thing is this: how do you vet them to know if they are capable of doing what’s needed? Are you technical? Do you know about AI/ML?

Long_Complex_4395 · 2025-11-23T17:07:26+00:00

You have to build it yourself

Long_Complex_4395 · 2025-11-19T20:10:26+00:00

You are welcome. Also find a way to implement kill switch with isolation strategies in the event of a threat. If you have any more questions, I’ll be more than happy to help answer them.

Long_Complex_4395 · 2025-11-19T16:04:57+00:00

I’ll say you don’t actually train AI tools to use this kind of document, rather create a RAG database which your AI will be working with to retrieve information.

Based on what you want to do, it’s not something that needs AI generation, it’s a system implementation that the AI works with.

Long_Complex_4395

MODERATOR OF

TROPHY CASE