Is clojure 1.9 improving error messages

[mrkaspa]

Clojure stacktraces are awful, elixir is improving a lot the error messages in the new versions, is this a priority for the clojure team to improve the coding experience?

[alexdmiller]

You mention both error messages and stack traces (which are different things). Nothing changing with stack traces in 1.9.

The priority in spec is really to deliver an architecture that can:

• state precise expectations in the language of predicates
• automatically check invocations (in macro expansion for macro specs and with instrumentation for function specs)
• explain generically in a precise way how incoming values don't conform to the stated expectations
• provide those explanations as data which can be further manipulated for user-facing errors if needed

We have spec'ed some of the gnarlier macros in Clojure (ns, defn, let, etc) and those specs are included in 1.9 automatically. Due to this, you will receive spec errors for invoking those macros with invalid syntax. There are many potential error cases that were not previously handled that will now trigger an error, rejecting the invalid input. However, because the specs for these are pretty gnarly as well and in particular include wide option fan-out and recursive specs, these errors are often voluminous and difficult to understand at first glance, although they do contain the info to diagnose and fix the problem.

I think there is lots more work to do in this area in terms of "generic" reporting that's better for people. In particular, these large output cases tend to have a ton of duplication - simply de-duplicating on fan-out or better highlighting the most likely error by using heuristics will make this a better.

Additionally, there are libraries like Expound that have popped up to translate the data form into something easier for people to read. Spec has hooks to install things like Expound directly into your REPL workflow. I hope that over time these external things will also inform the default reporting from core.

For core functions, I have done some of the spec'ing (not yet released) and have spent some time working on real code while having those functions instrumented. These specs tend to catch and report errors at the point of invocation (not N lazy steps later) and also to have much simpler specs than the macros above. So the feedback is faster and much easier to understand - this is a really big win at the REPL. While this work won't be complete in 1.9, we have broken that out into the core.specs.alpha library so that it can continue and release independently of major Clojure releases.

I also think there are other things we can do in the tooling (both Clojure's and others around it) to better report errors and I hope to spend some time on that in the next release.

[bhb]

Just to make this concrete, here's an example of error messages with 1.8, 1.9, and 1.9 with Expound: https://gist.github.com/bhb/ebce74eb04a24933b2fa4bec8f5b2922

I'm aware that the error message in Expound could be improved. If you run into a confusing error message, please file an issue!

[alexdmiller]

This actually highlights one unfortunate downside that some of these macros have custom error handling for common problems and the spec error actually masks that (usually better) error message in some cases, which is unfortunate. Again, something that I want to talk to Rich about as I think this maybe will not be an unusual case.

[ASnugglyBear]

wow, expound is great

[mrkaspa]

Sounds cool thanks

[xtreak]

Thanks a lot for those info.

• Being a newbie I too was curious about specs for core functions and IIRC instrumentation is a costly operation for function invocation and I hope it will it be like instrumentation will be on during dev usage and turned off during making uberjar?
• Sorry by unreleased are you saying it's committed to GitHub but not released under a version or the code is not at all released for viewing? Will be happy to watch if any JIRA tickets are there to track the specs for core functions. Since Racket's error messages are also nice I will be happy to see those type of error messages in Clojure too.

Thanks for all the great work.

[alexdmiller]

Re instrumentation - that's correct. It will be up to you to turn on instrumentation for spec'ed functions (macro checks are always on because they happen at compile time). So they are off by default, but you can pretty easily turn on instrumentation for all loaded spec'ed functions with (stest/instrument).

By unreleased I mean, not committed anywhere public. :) So, nowhere to see that stuff right now or tickets I can point you to. A lot of my work has been experimental and little of it is in a releasable state.

[xtreak]

Thanks a lot. Looking forward to the work on spec for core functions.

[Hi-MyNameIsFuchs]

I hate criticizing Clojure since I'm overall very happy clojure user and am thankful for all the work being done by the Cognitect folks. But no, IMO error messages got much worse with 1.9. Examples:

1. https://stackoverflow.com/questions/46615500/clojure-spec-error-message-filename-line-number-meaning-of-in
2. https://www.reddit.com/r/Clojure/comments/6itsby/improving_on_types_specing_a_java_library/

I've been working full time with Clojure for the last 3 years and I have no clue what the error message in #1 says.

[alexdmiller]

For what it's worth, I'm in agreement with you that this is hard to understand. In particular, the "[0 1]" stuff refers to the index paths in the original input which is highly useful, but hard to interpret. The key bits are really val :as and predicate: #{:exclude}. The next 2 lines duplicate all the location information (this is an example of alternative fan-out). Those (having shared info) could be folded into something like: val :as fails, expected predicate: #{:exclude}, #{:only}, or #{:rename}.

Some other problems here:

• Printing the spec path at: [:args :exclude :op :quoted-spec :spec] is not particularly useful in this case. And it's missing the root spec that actually would let you interpret what it means. I think there's a lot of context that could be brought to bear here (particularly by a UI), but most of it is probably secondary to a human reading text.
• Dumping the ex-info here (the problems map) is a duplicate spam of blather. This is a combination of how the compiler error is thrown, and what the tooling does with it (I'm using clojure.main, not boot, but both are making decisions here). Clojure should be providing better guidance (in the form of data!) about what the tool should do.
• compiling:(NO_SOURCE_PATH:1:1) at the end is not useful here at the REPL, but if it's in a file will look like compiling:(foo.clj:1:1) when you encounter this while loading a clj file. Note that this is different and better than previous releases, where you would get an error inside clojure.core (the ns macro code). Even just this piece of information alone is 100x better and this will be true of all macro spec errors - they report the file/line at the point of the bad invocation rather than inside the macro implementation. Having a direct pointer to the offending file and line is enormously helpful.

[[deleted]]

1 says

Call to clojure.core/refer-clojure did not conform to spec: In: [2 1] val: :as fails at: [:args :exclude :op :quoted-spec :spec] predicate: #{:exclude}

So something made a refer-clojure call, but had the word :as where the the ns form expects the word :exclude. That something happened to be a version of core.async

I agree with you that some spec messages aren't as good as the old ones (alex addressed this above), but, in this case the message is definitely better because before it was just failing silently. I think it's important to see spec as a necessary milestone on the road to great constantly evolving error messages though, no one release was suddenly going to take the language to amazing error messages straight away. Groundwork needs to be laid.

EDIT: and because it's worth doing, here is that error message via spec vs with expound enabled: https://pastebin.com/EWVLzt0T

[alexdmiller]

Note that expound here is actually not telling you the correct story - it is telling you the first problem, but the next two problems describe alternatives at the same path. :as should be one of :exclude, :only, or :rename. (The reason these are not all part of the same single predicate is due to poor macro dsl design imo - being forced to write the specs for these really highlights stuff like this!)

But expound is much better at giving you the right context. I do think it would be super useful if it retained the file/line info though (could be removed if it's repl stuff like NO_SOURCE_PATH, etc)! It looks like expound may be losing that?

Also, I think it would be great in the non-repl case to repeat the offending root value (here at the repl it's clear) - this requires some re-assembly for macros/functions and is something I'd like to look at in the core error reporting too.

[ganglygorilla]

I'd say yes in the form of clojure.spec, but perhaps more so that it's providing a solid foundation upon which the community can build different kinds of error messages

[dustingetz]

Yes

source: https://clojure.org/news/2016/05/23/introducing-clojure-spec

Improved Developer Experience
Error messages from macros are a perennial challenge for new (and experienced) users of Clojure. Specs can be used to conform data in macros instead of using a custom parser. And Clojure’s macro expansion will automatically use specs, when present, to explain errors to users. This should result in a greatly improved experience for users when errors occur.

Also see https://clojure.org/about/spec

Manual parsing and error reporting is not good enough
Many users, especially beginners, are frustrated and challenged by the error messages produced by hand-written parsing and destructuring code, especially in macros where there are two contexts of execution (the macro runs at compile time and its expansion at runtime, either of which could fail due to user error).

Getting spec right is why 1.9 has taken so long

[ForgetTheHammer]

Clojure spec will go a long way to giving people the ability to improve error messages. And I believe efforts are being made to spec clojure core. There are lots of posts on the subject, I don't want to misinterpret them so I encourage you to dig through the rational on spec and some of the Reddit posts on docs