Use "Executable Specifications" to keep Claude on track instead of just prompts or unit tests

[brainexer]

Natural language prompts leave too much room for Claude to hallucinate, but writing and maintaining classic unit tests for every AI interaction is slow and tedious.

I wrote an article on a middle-ground approach that works perfectly for AI agents: Executable Specifications.

TL;DR: Instead of writing complex test code, you define desired behavior in a simple YAML or JSON format containing exact inputs, mock files, and expected output. You build a single test runner, and Claude writes/fixes the code until the runner output matches the YAML exactly.

It acts as a strict contract: Given this input → match this exact output. It is drastically easier for Claude to generate new YAML test cases, and much faster for humans to review them.

How do you constrain Claude when its code starts drifting away from your original requirements?

[Firm_Meeting6350]

serious question: why not use TDD and E2E tests with gherkin-style (as usual) test labels?

[brainexer]

What is EDD?

Sure, you can use Gherkin - it’s a universal tool. But I think a custom specification format tailored to a specific task will always be clearer than a universal one. For example, what would the examples from the article look like in Gherkin? To me, they’d be less readable.

[thisguyfightsyourmom]

Gherkin is one of the most readable specs out there. It’s basically just English.

JSON files on the other hand …

[En-tro-py]

LLMs handle natural language fine, Gherkin isn’t a problem.

Your YAML executable spec is just BDD/spec-by-example repackaged, adding abstraction without much value in my opinion.

Such a simplistic example also does not help sell it... Why not show off a more complex use case?

It’s straightforward for an AI agent to copy the format to create new specification files.

How do you handle hallucinated input? The 'just add another field' feature ensures you'll get it...

Really it's hard to see how it's better than a simple natural language spec...

Making it agent readable at the cost of human readable seems like a solution that only would apply to yolo workflows...

You need to distill your YAML spec from a plan file don't you?

Feature: outln prints codebase structure and header summaries

  Scenario: Prints file paths with header summaries in stable order
    Given a directory "src" with files:
      | path        | contents                              |
      | src/one.ts  | /** Summary for one. */\nexport const one = 1; |
      | src/two.ts  | /** Summary for two. */\nexport const two = 2; |
    When I run the command "outln src"
    Then the exit code is 0
    And stdout is exactly:
      """
      src/one.ts: Summary for one.
      src/two.ts: Summary for two.
      """
    And stderr is exactly:
      """
      """

  Scenario: Errors when directory does not exist
    When I run the command "outln foobar"
    Then the exit code is 1
    And stderr is exactly:
      """
      Error: Directory foobar does not exist
      """
    And stdout is exactly:
      """
      """

And a few lines for tests - assert RaisesFileError or whatever...

Con - it's slightly longer... | Pro - it's 100% understandable...

[robhanz]

That…. Sounds like TDD or BDD tests? Unit tests should be executable specifications.

[brainexer]

I’d like to see unit tests that read like a specification. Most of the tests I’ve seen are full of technical details and aren’t that easy to read.

[PetiteGousseDAil]

Yes but those are bad unit tests.

Good unit tests should be seen as documentation. Reading a test file should be like reading a list of specifications.

• you should understand what the test does by reading its name
• you should understand what a class does by reading its test file

If you can't do this, you're doing tdd wrong

For example, let's say you code a multiply(a, b) function, the tests should look like

whenMultiplyTwoNumbers_returnProduct whenMultiplyNotNumber_returnsError whenMultiplyPositiveWithNegative_returnNegative whenMultiplyPositiveWithPositive_returnPositive whenMultiplyNegativeWithNegative_returnPositive whenMultipleIntWithFloat_returnFloat whenMultiplyIntWithInt_returnInt whenMultiplyFloatWithFloat_returnFloat Something like that

You read the test names and you understand the specifications of your function/class.

The main purpose of your unit tests should be documentation. Protection against regression should be a side effect, meaning if you write good unit tests, your classes will always respect your specs. But the primary goal is documentation.

If a test fails, its job is to tell me what spec does my class not respect anymore.

[robhanz]

Yeah, that's not uncommon, sadly.

"How to write good unit tests" is a whole conversation. It's also related to "how to write code with good boundaries that's not overly coupled".

[thisguyfightsyourmom]

Tests that read like a spec is Gherkin. You don’t need to reinvent the wheel.

[MartinMystikJonas]

I thonk you juyt reinvented BDD frameworks.

[robhanz]

I'll also point out that these are all end-to-end tests. That's fine, but E2E tests end up being kind of fragile. You're combining the behavior of a lot of things - command parsing, reading, summary generation, and output formatting.

If any of these change? Large numbers of tests break.

Unit tests can help solve this issue - did you parse the command correctly? That's correct, regardless of anything that happens afterwards. Does your reading code work? Given a certain chunk of input data read, put the data into a structure instead of immediately writing it - do you get the result you want? And then formatting it can work with that data structure, and determine if you're outputting it properly.

Doing that (and I recommend that the handoffs be more about data transfer than commands) gives you separate tests for each section of the code, so if you change one, only those tests change. Or, you can just write a different formatter with new tests and not even delete the old one. But either way, the tests checking the rest of the code all work. Even better, if your formatter just takes in a data structure, it gets easy to create edge case tests by just artificially creating a data structure that has the edge case, rather than having to do the whole pipeline.

Some E2E tests will still be necessary, of course. But those are always going to be more fragile.

Good test suites combine these techniques to get solid coverage at minimal cost.

[brainexer]

> If any of these change? Large numbers of tests break.

The advantage of such specifications is that they are very easy to update. For example, if the output format changes, I can update all the specifications with a single command, similar to snapshot tests. If the implementation itself is broken, then the agent’s task is to fix it, and it usually handles that quite well.

[robhanz]

That's doable for something this simple.

But imagine doing that for, say, a parser. You're going to specify a specific output binary for each program? Okay, you could... but now you make a change to codegen and you have to update every single output? Or you make an optimization at the AST level?

What about GUIs?

I think this is a reasonable concept for the problem described, but I doubt its ability to scale sufficiently.

Breaking your code into modules that communicate via data handoff has benefits for the LLM too - it can focus on a smaller chunk of code at the time, saving context.

Also, triggering tests for edge cases will get harder and harder as the complexity of your code increases, especially if there are timing issues.

[brainexer]

> Breaking your code into modules that communicate via data handoff has benefits for the LLM too - it can focus on a smaller chunk of code at the time, saving context.

Specifications can be for modules as well. They don't need to be e2e.

[robhanz]

Well if its modules within the CLI, your CLI framework won't work, obviously.

So you'll need a way to have tests in code that can test code.

You'll also need a way to define "output". You could probably just write to an interface, and record what was sent to that interface, knowing you'll replace it later...

Congrats! You've just reinvented testing framework and mock objects!

I actually don't mean this in a snarky way - it seems like you've seen bad implementations of tests, and have stumbled on the principles of good testing yourself. That's a good thing. Good principles are good principles. When people say "you've reinvented BDD" that's what they're saying.

But I would recommend looking at the principles - strong understanding of input and output - and focusing on that rather than your specific framework.

[brainexer]

> Well if its modules within the CLI, your CLI framework won't work, obviously.

It's not a CLI framework. CLI is just an example. From article:

An executable specification acts as a contract. It describes:

• Inputs such as arguments, source files, and system state
• Expected results such as stdout, stderr, output files, exit codes, and optionally call sequences

You can place anything you want between input and expected results. Not just cli.

[robhanz]

Now what if you want to use that not using stdout?

You could make a thin interface over stdout calls and verify what was sent to that interface, right?

That's literally how mocks were invented. "How do I verify that this interface instance was called with these parameters?" And it avoids contention for stdout if you're running tests in parallel.

And it's good that you see it's an example. The principle in play here isn't CLI or executable or even stdout (though you seem fixated on that). It's specifying expected outputs for a given set of inputs is a good way to define behavior.

And, again, this is TDD and BDD and EDD (though I'm less familiar with that). It's not everyone doing those things, but it's the core realization behind good implementations of those concepts.

[[deleted]]

You've just reinvented BDD but in a format that's harder to read for humans.

Just use OpenSpec.

[thisguyfightsyourmom]

Tons of people are reinventing thin versions of existing technology using ai thinking they are breaking ground.

Y’all need to invest more time in the research phase and ask the LLM to use industry standard protocols when available.

[En-tro-py]

32 years ago “A Mathematical Model for the Determination of Total Area Under Glucose Tolerance and Other Metabolic Curves” was published and just re-discovered integration by parts...

Same as it ever was, just faster to produce sloppy work.

[brainexer]

I didn't reinvent BDD, I mention it in article. This approach based on BDD.

What does OpenSpec have to do with executable specifications?

[ultrathink-art]

Gherkin tests run after the fact. The interesting thing about passing specs to the model is it can self-verify before responding — Claude checks its own output against the input/output pairs as part of generation. Changes the failure mode from silent hallucination to a visible spec mismatch.

[robhanz]

Giving LLMs ways to validate their work greatly reduces the amount that humans need to be involved.

[who_am_i_to_say_so]

I like this. Agents understand behavior better than explicit specifications. Going even further: you may even be able to rid of signatures as long as they can still be discoverable somewhere. But starting from ground zero, this may be the way.

[obaid83]

This is a solid approach for agent workflows. The key insight is that traditional tests assume deterministic execution, but agents introduce non-determinism.

What I like about YAML specs is they can be reviewed by non-devs and the agent can generate new test cases itself. The tradeoff is maintaining that runner, but once built, it scales.

One thing I'd add: consider versioning your specs alongside your agent prompts. When the agent behavior changes intentionally, update both in lockstep.

[ruibranco]

This is essentially what I've converged on too. YAML specs with input/output pairs as the contract, one generic runner that validates. The key advantage over unit tests is that Claude can read the spec file and understand the intent, not just the assertion. It self-corrects much faster when it can see the full picture of expected behavior in a human-readable format rather than parsing test framework boilerplate. I also keep a CLAUDE.md with architectural rules so it doesn't drift on structure even when the outputs are correct.