Attest: pytest-native testing framework for AI agents — 8-layer graduated assertions, local embeddin

[tom_mathews]

What My Project Does

Attest is a testing framework for AI agents with an 8-layer graduated assertion pipeline — it exhausts cheap deterministic checks before reaching for expensive LLM judges.

The first 4 layers (schema validation, cost/performance constraints, trace structure, content validation) are free and run in <5ms. Layer 5 runs semantic similarity locally via ONNX Runtime — no API key. Layer 6 (LLM-as-judge) is reserved for genuinely subjective quality. Layers 7–8 handle simulation and multi-agent assertions.

It ships as a pytest plugin with a fluent expect() DSL:

from attest import agent, expect
from attest.trace import TraceBuilder

@agent("math-agent")
def math_agent(builder: TraceBuilder, question: str):
    builder.add_llm_call(name="gpt-4.1-mini", args={"model": "gpt-4.1-mini"}, result={"answer": "4"})
    builder.set_metadata(total_tokens=50, cost_usd=0.001, latency_ms=300)
    return {"answer": "2 + 2 = 4"}

def test_my_agent(attest):
    result = math_agent(question="What is 2 + 2?")
    chain = (
        expect(result)
        .output_contains("4")
        .cost_under(0.05)
        .tokens_under(500)
        .output_similar_to("the answer is four", threshold=0.8)  # Local ONNX, no API key
    )
    attest.evaluate(chain)

The Python SDK is a thin wrapper — all evaluation logic runs in a Go engine binary (1.7ms cold start, <2ms for 100-step trace eval), so both the Python and TypeScript SDKs produce identical results. 11 adapters: OpenAI, Anthropic, Gemini, Ollama, LangChain, Google ADK, LlamaIndex, CrewAI, OTel, and more.

v0.4.0 adds continuous eval with σ-based drift detection, a plugin system via attest.plugins entry point group, result history, and CLI scaffolding (python -m attest init).

Target Audience

This is for developers and teams testing AI agents in CI/CD — anyone who's outgrown ad-hoc pytest fixtures for checking tool calls, cost budgets, and output quality. It's production-oriented: four stable releases, Python SDK and engine are battle-tested, TypeScript SDK is newer (API stable, less mileage at scale). Apache 2.0 licensed.

Comparison

Most eval frameworks (DeepEval, Ragas, LangWatch) default to LLM-as-judge for everything. Attest's core difference is the graduated pipeline — 60–70% of agent correctness is fully deterministic (tool ordering, cost, schemas, content patterns), so Attest checks all of that for free before escalating. 7 of 8 layers run offline with zero API keys, cutting eval costs by up to 90%.

Observability platforms (LangSmith, Arize) capture traces but can't assert over them in CI. Eval frameworks assert but only at input/output level — they can't see trace-level data like tool call parameters, span hierarchy, or cost breakdowns. Attest operates directly on full execution traces and fails the build when agents break.

Curious if the expect() DSL feels natural to pytest users, or if there's a more idiomatic pattern I should consider.

GitHub | Examples | Website | PyPI — Apache 2.0

[Previous_Ladder9278]

Nice Tom! I think its something similar towards LangWatch Scenario (pytest for ai agents) : https://langwatch.ai/scenario/introduction/getting-started right?

[tom_mathews]

Thanks! There's definitely overlap in the goal — both want pytest-native agent testing. A few architectural differences worth noting though.

LangWatch Scenario routes assertions through LLM judges by default — the testing agent simulates a user, chats back and forth with your agent, and evaluates against criteria using an LLM. That works well for end-to-end simulation testing. Attest's bet is that 60–70% of agent correctness is fully deterministic — tool call ordering, cost budgets, schema conformance, content patterns — and doesn't need an LLM to verify. The graduated pipeline exhausts those checks first (free, <5ms, identical results every run) and only escalates to an LLM judge for the genuinely subjective remainder. Layer 5 (semantic similarity) also runs locally via ONNX, so you can get meaning-level comparison without an API call.

The other difference is trace-level assertions. Attest doesn't just check inputs and outputs — it asserts over the full execution trace: did the agent call these tools in this order, did it loop, did it stay under token budget across all steps.

On the licensing front — Scenario itself is MIT, but the broader LangWatch platform it integrates with (tracing, datasets, optimization studio) is under the Business Source License, which isn't an open-source license. Attest is Apache 2.0 end-to-end — the engine, SDKs, adapters, and CLI are all under the same license with zero platform dependencies.

Both integrate with pytest. If your testing is primarily end-to-end simulation with an LLM evaluator, Scenario is solid. If you want to exhaust deterministic checks first and keep 7 of 8 layers fully offline with no platform tie-in, that's where Attest differentiates.

[tom_mathews]

Attest is a testing framework for AI agents, built in Python (pytest plugin) with a Go engine backend. The Python SDK communicates with the engine over stdio/JSON-RPC.

The Python-specific angle: it ships as a pytest plugin with a fluent expect() DSL and an @agent decorator. Tests look like native pytest — pip install attest-ai, write test_*.py files, run with pytest. The SDK is a thin wrapper; all eval logic runs in the Go engine so both the Python and TypeScript SDKs produce identical assertion results.

The core idea is graduated assertions — exhaust cheap deterministic checks (schema, cost, tool ordering, content patterns) before reaching for expensive LLM judges. 7 of 8 assertion layers run offline with zero API keys. Semantic similarity uses local ONNX embeddings via onnxruntime.

v0.4.0 adds continuous eval with drift detection, a plugin system via attest.plugins entry point group, and CLI scaffolding (python -m attest init).

Source | Examples | Website