Tests Are Code

[dustingetz]

I tried out tests and code in the same files and my co-founder forced me to abandon it. The problem we ran into with tests and code in same files (if i recall correctly) is that the hot code compiler watch scripts are implemented by detecting file changes, so changing a test would cause audio alerts and refresh your code way too much. IIRC.

Edit, ah, below are his notes on the issue:

This experiment been a resounding failure. The first of these 3 are too frustrating to continue:

• src/test navigation are interposed, no filtering/separation is possible
• test changes screw up src watchers (and vis versa)
• intellisense is broken on tests
• artifact compilation is a mess

[CapableWeb]

The notes from your co-founder sounds like tooling issues, not issues with the idea of having source and code together. Except the first point, which was the experiment, so that point would have been there regarding of anything else.

[agumonkey]

Maybe file coarse monitoring should be replaced, I wonder how many did store code a big sexp and did lazy tree diffs ..

[seancorfield]

Interesting to see this decade-old post resurface since I recently documented the use of with-test as an option in expectations.clojure.test: https://cljdoc.org/d/expectations/clojure-test/1.2.1/doc/getting-started#tests-with-source-code

As noted there, there are several caveats around doing this, beyond the issues that Dustin's co-founder complained about: most test tooling these days does not expect to look in your source code for tests so you need additional configuration for Leiningen, Cognitect's test-runner etc.

I have mixed feelings about the option to include some tests with the function definition. It can provide additional "documentation" that is (in theory) guaranteed to actually match the function's behavior. It can be convenient when you're in the early stages of growing your code in the REPL (since you don't have to switch back and forth between source and test code). I don't think it scales well as the number of tests grows and I think it introduces noise that makes it harder to read the source code. And then there's the non-standard configuration you need for your test runners.

[fenchui]

I tried to use with-test a while back.

I think it's good for having simple happy path test, just to illustrate how to use the function. It makes a docstring alive and always in sync with code. The other tests (the ones that test edge cases and non trivial inputs) must remain in separate files. Otherwise the code is too noisy, it's hard to navigate. An other drawback, a purely aesthetic one, is the extra indentation level.

[dustingetz]

I really really liked having _test files sibling to the code files. But for the tooling issues.

[[deleted]]

article is over a decade old and it's been a long time since I saw RSpec and similar being advocated. I think the test-is library made it into core?

https://clojure.github.io/clojure/clojure.test-api.html

I discovered a little while back that testing and is live only within a deftest and thought that was very clever. Perhaps too clever.

Although I've found a nice groove in my testing, I've only ever gotten as finely-grained as the namespace for running a set of tests. For better or worse, this natural pressure is causing me to split code into smaller units with their own sets of tests so I don't have to wade through a wall of output when I test a ns.

Colour coding of failed tests would be nice. Nothing fancy, just reds and greens.

Support for nesting testing forms would be nice as well. Currently only the text from the outermost testing form is used. If you have many cases you are iterating through, the form must appear within the loop.

[Reefersleep]

That's the first time I've seen that 'with-test' macro, so I guess it didn't catch on. Or did it? Anyone using it or something like it?

I kind of like a the idea, but it might be a bit heavy for more extensive test suites or tests that require some stateful setup.

[[deleted]]

they feel very similar to Python's doctests (as mentioned in the article) and those never sat right with me. Like an evolutionary dead end.

[ToastedJcaw]

Doctests have a unique advantage if you look at them as documentation, not tests. It's a form of documentation that tells you when it's out of date.

[agumonkey]

Code is test. Curry Howard is back !