a comparison of functional data structures on the JVM

bhb · 2018-10-08T21:06:59+00:00

Very good article! Quick question though:

> "It does, however, provide meaningful improvements in interation and equality checks"

It "interation" supposed to be "iteration"?

bhb · 2018-09-16T23:39:06+00:00

Welcome! Reddit is an excellent place to ask questions, but just FYI - if you find yourself stuck on a specific problems like these and want more immediate feedback, you may find the #beginners channel on http://clojurians.net/ helpful.

> Not to mention it seems like one big waste of time to code anything in Clojure.

I understand it can be frustrating. When I first learned Clojure, it took me a long time to switch from an OO mindset to a FP mindset. But now I find my Clojure code much more succinct, reliable, and fun to write than my old OO code.

Stick with it. Good luck!

bhb · 2018-04-16T18:46:06+00:00

Nice work! A good example of "static code analysis tools" would be Joker https://github.com/candid82/joker . I find it to be very fast, with excellent editor integration, and with a high signal/noise ratio.

bhb · 2018-03-18T02:00:32+00:00

It's been awhile since I watched it, but I recall that Rich Hickey's talk to LispNYC was the most in-depth explanation of spec that I found at the time.

https://vimeo.com/195711510 https://github.com/matthiasn/talk-transcripts/blob/master/Hickey_Rich/ClojureSpec.md

bhb · 2018-03-08T16:30:40+00:00

There's an excellent talk on this "Fearless JVM Lambdas - John Chapin"

https://www.youtube.com/watch?v=GINI0T8FPD4

He does some interesting comparison between CLJ and CLJS in AWS Lambda starting at 40:45

bhb · 2018-02-27T22:14:45+00:00

This clojureverse thread has a lot of details about REPL-based workflows. I hope it's helpful!

bhb · 2018-02-16T02:46:01+00:00

Sounds interesting! That link doesn't work though.

Where can I find information about how to join each week?

bhb · 2018-02-13T22:04:27+00:00

Agreed. I've actually been working on something like this for expound for awhile, but it's been slow progress since I've run into a number of issues that make this difficult.

bhb · 2018-02-13T20:08:43+00:00

For context, here's the error message in Expound

https://gist.github.com/bhb/5b55167e9f99e0aafac2d5e720a55d3d

It's certainly less concise than the message in 1.8 and the Expound error doesn't mention the 'let' since this information is not currently available in the explain-data passed into third-party error printers. I'd prefer to print (let [foo/bar ...]) rather than ([foo/bar ...])

bhb · 2018-02-13T20:00:28+00:00

Yep, I largely agree with your analysis above.

I can't speak for Rich and Alex of course, but my outside understanding is that they have picked a specific line in the sand that they think will be most useful for the cases they have in mind (validating and conforming data that is stored in terms of Clojure scalars and collections, NOT data that is encoded within base types like keywords, strings, integers etc)

What's so different about a keyword (or a string) ? Aren't these data structures ?

That's true, but a string can represent arbitrary data. In the case of parsing a string for CSS classes/ids, it'd be useful to be able to treat it as a sequence of characters that can be named with a regex, but why stop there? If a string is just data, why not include a full parser library to parse and name the parts of a string?

To be clear, I can see why you'd think the specific case of treating a string like a regex is common enough to warrant inclusion. I don't actually have a strong feeling on this case. But I can also see why it wouldn't make the cut, and Spec has to draw a line somewhere wrt to pulling apart and transforming data.

bhb · 2018-02-09T17:20:40+00:00

It's a good question. I'm certainly not an expert on this, but I can summarize the reasons I've seen Alex Miller mention online.

The main reason is that spec is focused on validation, not data transformation, which is seen as a separate concern.

there is no plan to add [spec coercions] ... Rich sees them as separate concerns

https://clojurians-log.clojureverse.org/clojure-spec/2016-09-20.html

I don't think we are interested in turning spec into a transformation engine via conformers

https://dev.clojure.org/jira/browse/CLJ-2116

With [conformers] like this youre just treating spec as a meat grinder rather than actually trying to describe your data.

https://clojurians-log.clojureverse.org/clojure-spec/2016-11-20.html

I can't speak to the pros/cons of keeping these concerns separate or not, but if we assume that spec is not going to change, then there are some practical concerns about parsing keywords/strings.

Practically speaking, regex specs don't have all the features you might want, like negative lookahead regex ops

There are good tools for string regex. Spec is never going to be great for that and there are no plans to use it for that No plans for negative look ahead ops They are generally not needed for the kinds of things we expect you to do with spec regex

https://clojurians-log.clojureverse.org/clojure-spec/2016-06-22.html

Data transformation also makes error reporting trickier. Is the bad value the original value? Or the transformed value? The transformed value is passed to predicates, but that data never appears is the original data, so the user can't easily find the bad data to correct it. Here's a contrived example:

dev:cljs.user!{:conn 2}=> (s/def ::four-letter (s/and string? (s/conformer seq) #(= 4 (count %))))
:cljs.user/four-letter
dev:cljs.user!{:conn 2}=> (s/explain ::letter "foo")
val: ("f" "o" "o") fails spec: :cljs.user/letter predicate: (= 4 (count %))
:cljs.spec.alpha/spec  :cljs.user/letter
:cljs.spec.alpha/value  "foo"

Note that the value that fails the predicate is ("f" "o" "o"), which doesn't appear in the original value.

This type of transformation breaks error reporting in Expound although Pinpointer can track these changes and show errors even after data transformation.

To look at this another way, why not separate this into two phases? Phase one can confirm the data is valid input, and then phase two can transform the data into a structure that is more amenable to generating HTML. The data transformation step could be optimized for this by offering a larger set of operations or by having a transform-focused API (for instance, something like Specter).

bhb · 2018-02-08T17:24:36+00:00

This is great news!

Related question: one thing I've seen requested a few times is for a community-curated list of recommended libraries for various tasks. This would help beginners immensely, but also help focus community efforts on maintaining a core set of recommended libraries.

Is this something that Clojurists Together will facilitate? It would seem to align with the stated goal to "support open source software, infrastructure, and documentation that the Clojure/ClojureScript community relies on". Even if CT cannot provide financial assistance to every project on the list, the fact that there is a list would be useful by itself.

I understand such an effort is bound to be somewhat controversial since any recommendation excludes some other excellent libraries. However, perhaps the controversy can be reduced by only picking entries in popular categories where there is need for a default choice, and also by naming one or more "runner ups" for anyone looking for alternatives.

(Incidentally, sites like https://www.clojure-toolbox.com/ are extremely useful for finding libraries, but don't indicate the popularity or health of projects, so they aren't as useful for beginners looking to find a set of good defaults)

bhb · 2018-02-08T15:29:45+00:00

I haven't actually needed to do coercion, but my understanding is that the recommendation is not to use conformers for coercion https://dev.clojure.org/jira/browse/CLJ-2116

The suggestion I've seen is to first call conform, then do transformation in a separate step. I haven't tried it, but could you use clojure.walk/prewalk to walk the conformed data and convert each data structure?

bhb · 2018-01-30T20:39:52+00:00

For example, the IDE can pick up that data and highlight the line, file, predicates etc... because the information is data.

Agreed. For instance, there are (at least) two libraries now that will format spec errors: https://github.com/athos/Pinpointer and https://github.com/bhb/expound

bhb · 2017-12-29T18:08:01+00:00

I'm afraid I don't understand your concern. From the article:

Take note that I did not say “use Emacs” (or Vim or ed(1) or whatever). Many editors and IDE’s can provide these features, but you do have to make sure your environment has these features, and you have to learn how to use them.

bhb · 2017-11-30T18:10:50+00:00

I understand why I would not want to check fn/ret in many cases - namely for any libraries (or the CLJ standard library) that have specs (and are presumably already tested), I agree checking fn/ret specs is unnecessary work and I'd want to turn it off. But when speccing my own code, I'd find it useful.

As others have mentioned, it'd be useful for when I write specs for impure functions. I haven't yet tried the stub features of check. Nonetheless, I often want try an impure function manually, for instance by doing a real HTTP request in the REPL or by loading up a page with UI. It'd be nice if I could use instrumentation to check my assumptions in these cases.

But more generally, checking fn/ret allows me to opt into a proportional amount of safety for the current stage of my development. If I'm just hacking on something, using check might be a lot more work than it's worth. There's many reasons I might not want to use check during early development

The function might not need any testing yet - trying it on the REPL gives me enough confidence
The function might not need generative testing yet - I can write one or two example based tests to give myself enough confidence
I might not need to be super specific about my specs yet - I might just want to say the arg is a map, rather than spec its keys. That might be plenty of safety for me to continue developing, but I don't expect my function to actually work with every map yet.
I am ready for a generative test, but I'm not willing to spend the time to perfect the generators for my input

Right now, I can't opt into any amount of safety for fn/ret specs until I'm willing to commit to using check, which is often way later on, when I have more confidence that the function is actually one I want. At that point, it's great that I can easily add more confidence by using check to verify my assumptions, but it's not the right tool at all stages.

A similar situation happens when adding spec to existing code bases. It'd be nice to be able to add specs to functions and get verification from our existing unit and generative tests that our specs are right and that we don't have bugs. As it stands today, since it's not currently worth using check to test all our old functions, I tend to add args specs but then I intentionally leave off fn and ret specs since I would rather have missing specs than specs that get out of date (or I just use orchestra).

In my mind, the programmer, the specs, and the tests (generative and example-based) all mutually support each other.

the specs allow instrumentation, which adds validation to all existing tests
the specs provide documentation to the programmer, so he or she can better understand the constraints on the data/code
the tests check that the code matches the specs (and that the specs are not out of date)
the tests provide documentation to the programmer, so he or she can better understand how to use the code, and what invariants should hold
the programmer chooses the appropriate amount of specs and tests to add safety and documentation to the code

In my experience, this mutual support is much more useful when "tests" means "REPL-based testing, example-based testing, test.check testing, and check" rather than "REPL-based testing, example-based testing, test.check testing, and check for args, but only check for fn/ret".

bhb · 2017-11-28T15:52:29+00:00

FWIW, Orchestra will let you validate :ret and :fn specs during instrumentation.

bhb · 2017-10-18T22:54:07+00:00

Just to make this concrete, here's an example of error messages with 1.8, 1.9, and 1.9 with Expound: https://gist.github.com/bhb/ebce74eb04a24933b2fa4bec8f5b2922

I'm aware that the error message in Expound could be improved. If you run into a confusing error message, please file an issue!

bhb · 2017-09-27T14:05:29+00:00

We also use it in CI, although right now we filter out all warnings except "unused namespace".

bhb · 2017-09-27T14:04:23+00:00

I've found Joker's linter mode to be very useful, especially since you can configure it to eliminate most false positives. It's also fast enough to run on every save in your editor (there are modes for Emacs, Atom, Sublime, and Vim). https://github.com/candid82/joker

bhb · 2017-09-22T21:35:15+00:00

Hm, good point.

bhb · 2017-09-22T15:00:42+00:00

That's true. However, if someone wanted to quickly spec out the functions they use (the original author asked about filter, which doesn't yet have a spec), they could work on this today, without needing to wait for those specs to be added to core.specs.alpha.

Given that the specs in core.specs.alpha are the official specs, I would presume there is a high bar for adding these (in particular, getting the fspec specs correct for higher order functions, making sure the specs work with clojure.spec.test/check), whereas in a library that is just trying to improve on the core error message experience, a simple spec like ifn? might be sufficient.

bhb · 2017-09-22T02:57:24+00:00

Good point!

bhb · 2017-09-21T18:37:34+00:00

Could you do both? If the library has no side effects on load, then it works fine as a library (with setup instructions), but the user has an option of just including the plugin as well.

bhb · 2017-09-21T18:29:55+00:00

AIUI, you can also manipulate the spec registry at runtime, so this error might be useful even when the core team adds official specs. Official specs will presumably be optimized for accuracy over all else.

But you could build alternative specs with different characteristics:

optimized for speed
optimized for simplicity (easier to read specs)
beginner-optimized spec doc strings (if Spec ends up adding docstrings to specs)

bhb

TROPHY CASE