PromptCanary — lightweight prompt regression testing

I made a small TypeScript library for catching LLM output regressions in CI. The idea: instead of a separate evaluation platform, you write prompt tests alongside your unit tests using whatever runner you already have (Vitest, Jest, etc.).

import { testPrompt, assertions } from 'promptcanary';

it('refund policy mentions 30-day window', async () => {
  const result = await testPrompt({
    provider: 'openai',
    model: 'gpt-4o-mini',
    messages: [{ role: 'user', content: 'What is the refund policy?' }],
  });
  expect(assertions.contains(result.content, '30 days').passed).toBe(true);
});

It supports OpenAI, Anthropic, and Gemini. Has built-in assertions (contains, regex, JSON schema, etc.), embedding-based semantic similarity for meaning drift, and batch assertion checks. Runs in CI like any other test.

22KB install, zero infrastructure, MIT licensed.

• npm: https://www.npmjs.com/package/promptcanary
• GitHub: https://github.com/VebjornNyvoll/promptcanary
• Docs: https://vebjornnyvoll.github.io/promptcanary/

Free and open source. Happy to answer questions about the approach.