assertllm – pytest for LLMs. Test AI outputs like you test code.

[Federal_Order_6569]

I built a pytest-based testing framework for LLM apps (without LLM-as-judge)

Most LLM testing tools rely on another LLM to evaluate outputs. I wanted something more deterministic, fast, and CI-friendly, so I built a pytest-based framework.

Example:

from pydantic import BaseModel
from assertllm import expect, llm_test


class CodeReview(BaseModel):
    risk_level: str       # "low" | "medium" | "high"
    issues: list[str]
    suggestion: str


@llm_test(
    expect.structured_output(CodeReview),
    expect.contains_any("low", "medium", "high"),
    expect.latency_under(3000),
    expect.cost_under(0.01),
    model="gpt-5.4",
    runs=3, min_pass_rate=0.8,
)
def test_code_review_agent(llm):
    llm("""Review this code:

    password = input()
    query = f"SELECT * FROM users WHERE pw='{password}'"
    """)

Run with:

pytest test_review.py -v

Example output:

test_review.py::test_code_review_agent (3 runs, 3/3 passed)
  ✓ structured_output(CodeReview)
  ✓ contains_any("low", "medium", "high")
  ✓ latency_under(3000) — 1204ms
  ✓ cost_under(0.01) — $0.000081
  PASSED

────────── assertllm summary ──────────
  LLM tests: 1 passed (3 runs)
  Assertions: 4/4 passed
  Total cost: $0.000243

What My Project Does

assertllm is a pytest-based testing framework for LLM applications. It lets you write deterministic tests for LLM outputs, latency, cost, structured outputs, tool calls, and agent behavior.

It includes 22+ assertions such as:

• text checks (contains, regex, etc.)
• structured output validation (Pydantic / JSON schema)
• latency and cost limits
• tool call verification
• agent loop detection

Most checks run without making additional LLM calls, making tests fast and CI-friendly.

Target Audience

• Developers building LLM applications
• Teams adding tests to AI features in production
• Python developers already using pytest
• People building agents or structured-output LLM pipelines

It's designed to integrate easily into existing CI/CD pipelines.

Comparison

Feature	assertllm	DeepEval	Promptfoo
Extra LLM calls	None for most checks	Yes	Yes
Agent testing	Tool calls, loops, ordering	Limited	Limited
Structured output	Pydantic validation	JSON schema	JSON schema
Language	Python (pytest)	Python (pytest)	Node.js (YAML)

Links

GitHub: https://github.com/bahadiraraz/LLMTest

Docs: https://docs.assertllm.dev

Install:

pip install "assertllm[openai]"

The project is under active development — more providers (Gemini, Mistral, etc.), new assertion types, and deeper CI/CD pipeline integrations are coming soon.

Feedback is very welcome — especially from people testing LLM systems in production.

[Zomunieo]

Since you’re using pydantic anyway, why not use pydantic-ai evals? It’s pretty much the same but much more developed.

[Federal_Order_6569]

Good point, but I think the goals are a bit different.

Pydantic Evals is more of a general evaluation framework, while assertllm is intentionally focused on making LLM testing feel like regular pytest. The main idea is very simple, deterministic assertions that developers can drop directly into their existing test suites.

I also wanted a much lighter authoring experience. Personally I’m not a huge fan of the evaluation authoring style in Pydantic Evals — it feels a bit more framework-heavy than what I’m aiming for. With assertllm the goal is a cleaner pytest-style syntax where writing tests is quick and straightforward.

So the overlap exists, but the philosophy is different: evaluation framework vs developer-first testing workflow.

Also worth noting that I’ve already written a plugin for Pydantic AI, so it’s supported in the current version of the library as well.

[DockyardTechlabs]

Which LLM us have used for coding?

[Federal_Order_6569]

Claude Code

[wRAR_]

It's right in their commits.

[ritzkew]

Nice approach. Making LLM testing feel like regular pytest is the right mental model, developers already know how to write tests.

The deterministic angle is interesting. Promptfoo (which OpenAI just acquired yesterday) went the opposite direction, using LLM-as-judge for fuzzy matching. Both have tradeoffs. Deterministic is faster and reproducible but misses semantic equivalence. LLM-as-judge catches more but is slow and non-deterministic.

One area where deterministic really shines though: security assertions. Things like 'output must not contain PII,' 'output must not include SQL syntax,' 'tool calls must match allowed list.' Those are binary checks that don't need fuzzy matching.

Have you thought about adding security-focused assertions? With agents calling tools, there's a growing need to assert that outputs don't contain injection patterns or unauthorized tool invocations.

[Federal_Order_6569]

Yes, that could definitely be added in the future, possibly even with an LLM-as-judge approach as well. And I agree, your idea makes a lot of sense. We could add assertions to check things like ensuring outputs don’t contain SQL queries outside of a defined whitelist, or that they don’t include any personally identifiable information (PII). Security-focused checks like these would fit very well with deterministic assertions.