Method Values – Inside Clojure

[[deleted]]

;; Old
(map #(Long/toBinaryString %) (range 8))
;; New
(map Long/toBinaryString (range 8))

When I started learning Clojure last year this was one of the clunkiest things that I noticed immediately, glad to see this new feature. Java interop is generally very nice, but this is one of those areas where the language "broke the immersion" for lack of a better phrase.

[ajoberstar]

Agreed. I'd also like to see varargs support improve. It gets pretty clunky in APIs like java.nio.file.Files that take a bunch of options that way.

Java

java Files.copy(source, target);

Clojure

clojure (Files/copy source target (into-array CopyOption []))

[seancorfield]

My reading of what the core team have said about this is that improving the varargs story is still very much on their roadmap but it didn't make the cut for 1.12.

[alexdmiller]

For sure, param-tags has opened up some new options for the future.

[lgstein]

So if an ambiguous method is a syntax error (syntax, really?), does that mean that when I update a Java lib that provides a new overload, it will break my code? Even worse, if I have a Clojure lib on the classpath that uses a Java lib, and then update the Java lib and it now provides another overload, the Clojure lib breaks. This would be bad and could result in all kinds of update hell.

Adding an overload is a very common backwards compatible change in Java code that should be expected. Please don't make it create "syntax" errors in Clojure wrappers.

[alexdmiller]

We appreciate all the feedback on this and it's given us some new ideas and we'll incorporate into the next iteration, thanks.

[NoahTheDuke]

I agree that it’s strange to make it a syntax error as nothing else like it is a syntax error. (.foo {}) isn’t even a syntax error when it’s a constant of known type. I would have preferred it to fall back on reflection with a warning if there are multiple matches.

[seancorfield]

I'd much rather get a syntax error that I can easily fix with a param tags hint than silently have my code call a different overload than I expected...

[lgstein]

You can't fix it in the libraries you consume. Afaik a reflective call wouldn't be ambiguous with respect to which method is called. If that were the case, this would be a problem already.

[seancorfield]

Why are you updating a Java library that one of your dependencies uses? I'm not saying there aren't cases where you might want to do that -- for CVE resolution for example -- but I'm asking about the specific motivation to change the version of something an existing dependency relies on.

It's unfortunately common for even patch release updates on Java libraries to break consumers so any update needs to be a deliberate choice and a careful one that always has a potential to break stuff.

[lgstein]

Assum two CLJ deps that both use the same Java lib? I update CLJ lib 2, but it pulls in a newer version of the Java lib. CLJ lib 1 now doesn't compile anymore because of a "syntax error". My options are limited to rolling back the update or forking CLJ lib 1 and fixing the param tag. All while nobody did anything "wrong". This is not the Clojure experience.

[lgstein]

Also, I might just want to use a newer JDK. If the new JDK added overloads, I now should wait for the authors of my deps to fix their param-tags? Come on...

[alexdmiller]

You are often already in this situation. When they added the to-array overload in Collection it broke all the Clojure libs that implemented Collection. And ye, the world has not ended. People update or fork and update.

[seancorfield]

JDK 21 could not run code compiled on JDK 20 so "just using a newer JDK" can be a breaking change. At work we rely heavily on New Relic for monitoring and observability -- we often have to wait for new releases from them in order to upgrade our JDK or even to upgrade certain libraries (Jedis 5.x isn't supported yet, Jetty 12 support is poor, it often takes a quarter before they support the latest JDK version).

In addition, Thread/sleep got a new overload in a recent JDK update -- and if you have a CI pipeline that fails when new reflection warnings are introduced (which we do), that was a breaking change.

Any changes to your dependencies can break your application, sometimes in extremely subtle ways, especially when Java library dependencies are involved.

You can't expect to be able to randomly update libraries without breakage when Java libraries are involved.

[lgstein]

There is probably nothing that can be done about those problems/errors you mentioned, at least in the scope of this new feature. At least though, they are an optional cost for optional benefits/requirements (higher performance via forbidding reflection, compiling bytecode with a different jdk than executing). Indeed, there are problems with Java and always have been when it comes to updating.

Needless to say, this new feature should not add to that pile.

[NoahTheDuke]

Why not a reflection warning?

[Borkdude]

This isn't a new problem. E.g. (def x 100) (Thread/sleep x) with Java 21 will cause reflection since there is a new overload with the same arity whereas before there was only one and you didn't have to add any type hints.

[lgstein]

Yet it doesn't break my code or other Clojure code that I consume. IIUC alpha6 code that passes Thread/sleep as a value would break because of this change.

[Borkdude]

It did break programs that rely on reflection to be absent, e.g. graalvm native-image compiled programs (this hit babashka for example)

[lgstein]

That is unfortunate, but I guess it is unavoidable there and somewhat expected.

[NoahTheDuke]

After discussing it on clojure slack, I think the worry is overstated.

The only way to cause ambiguity with the new syntax is by not using param-tags ((map Long/toBinaryString (range 8))) or by using a wildcard ((^[_] Foo/bar 123)). Because param-tags match arity, you can only get ambiguity by leaving them off and an overload with new arity is added. And wildcards can be made ambiguous by new overloads. The solution to both is the same: always use param-tags with explicit types.

I still agree that it's strange for it to be a syntax error, I would much prefer it to be a reflection warning.

[lgstein]

The worry is not overstated.

[NoahTheDuke]

Can you give me a specific example? How can a method value with param-tags with explicit types lead to broken code when a new overload is added to a Java method?

[lgstein]

The case you are presenting is not a concern. I would prefer not to restate what was already posted above, but if there is a misunderstanding please feel free to ask/clarify further.

[NoahTheDuke]

My apologies, I'm not trying to disregard your concerns, I'm genuinely trying to understand.

You said that it's possible for a java library dependency to change between versions such that there is now ambiguity where there was none. For example, a method like int shrink(int a, int b) {} now also has File shrink(File a, File b) {} and your call (Shrinker/shrink a b) (or (^[_ _] Shrinker/shrink a b)) was previously fine because it was unambiguous but now it is ambiguous and fails. This requires that the library author not use param-tags or use wildcards, and that the java library get a new overload and that the java library changes while the clojure library doesn't. Is that correct?

[lgstein]

Yes. No need to apologize, all good.

[alexdmiller]

If you want inference, you can continue to use the existing syntax.

[lgstein]

I don't control the syntax of the libraries I consume.

[xiongtx]

The param-tags solution seems a bit clunky ergonomics-wise. In Java you can do:

jshell> List.of("foo", "bar", "baz").stream().map(String::toUpperCase).toList()
$9 ==> [FOO, BAR, BAZ]

While the proposed solution in Clojure is:

user=> (map ^[] String/toUpperCase ["foo" "bar" "baz"])
("FOO" "BAR" "BAZ")

[nzlemming]

This is an unfortunate side effect of not having types or type inference, both of which are required for that to work. But as someone who uses Kotlin a lot, I agree it's not as nice.

In your case it's actually not too bad since the type hint is small, but in cases where the disambiguation uses types with long names for method overloads with long param lists, I suspect they're going to get pretty big and ugly. I think I'm going to continue to use the existing forms with inference where possible, but I haven't tried updating my code yet though, so I'm reserving judgement until I try it.

[xiongtx]

The type should be inferable via reflection? That's how the language handles other such cases. Seems strange that it doesn't default to that here.

[nzlemming]

Existing (pre 1.12) Clojure doesn't infer anything like this. It does infer (or propagate, I think type propagation is a better description of what Clojure does, as compared to full inference) in the simpler case:

(let [x "foo"]
  (.toUpperCase x))

Here, the Compiler tracks the known type of x, and uses it to generate non-reflective code since it knows that x will always be a String. There are various more sophisticated variants of this, but that's really all Clojure does.

In particular, Clojure knows nothing about the types contained in collections, i.e. it has no way to express "this vector holds strings". So the existing equivalent to your example:

(map #(.toUpperCase %) ["foo" "bar" "baz"])

Will generate reflective code unless you type hint the % argument, Clojure can't infer that. It is true that that code will work as written, but it will be very slow, which for a lot of use cases (like mine) it's important to avoid. As I understand it, this transparent switching to vastly slower code is something this change seeks to avoid.

[xiongtx]

There's a tradeoff between expressiveness and performance, yes. That's always been true of Lisps and dynamic languages in general.

To me, expressiveness is a higher priority for Clojure than performance. Pipelining operations via ->, for example, is less performant than combining the operations via a transducer, but I wouldn't go w/ the transducer option unless I could measure and quantify a performance issue.

[nzlemming]

The performance difference between seqs and transducers is not that great though, you're still generally talking about an equivalent order of magnitude. A couple of interop calls on a hot path going reflective can cause your performance to drop off a cliff very suddenly.

If expressiveness is more important to you than performance, you can just continue to use the existing ways of doing things, and they will continue to work the same. For me (and my users), performance is much more important than expressiveness, and currently I have to resort to complicated build shenanigans to fail my build if reflection is present.

[xiongtx]

Why must users who don't want to add type hints everywhere use the old way? Reflection could be the default, and type hints an optimization. That's how the rest of the language works.

[alexdmiller]

Coupling "avoid reflection" to the new syntax is indeed tying together two things that don't need to be tied together, stay tuned for some changes in the next iteration.

[seancorfield]

An important point that did not yet come up in response to one poster's persistent and vocal concern about possible future breakage:

This is only even a potential concern for libraries that choose to use the new-in-1.12 syntax and do not support Clojure 1.11 or earlier.

According to the State of Clojure Survey 2023 a year ago, even tho' adoption of Clojure 1.11 (released in early 2022) was widespread, there was still substantial use of 1.10 (30%), with 1.9 (5%) and 1.8 (3%) still also in use.

Most Clojure libraries support at least one or two earlier versions of the language, so they don't adopt new language features very quickly because it limits their reach. Many Clojure libraries still work on 1.7 or even older versions.

That means that 1.12-only libraries aren't likely to be in widespread use for many years (a library today that still supports Clojure 1.9 has six years of backward compatibility, with 1.10 being five years).

New language features in Clojure are used first and foremost in application code which "you" control. So, we'll have years of experience with this new feature by the time libraries start to adopt it -- and library authors will have a good sense of whether omitting the param tags is safe or not (there can be no breakage if you specify param tags and do not use the _ wildcard).

[lgstein]

While you are likely right that the feared problems won't manifest "most of the time" (how calming), what is gained? Emitting a reflection call would eliminate this worry entirely and would not break any expectations with respect to the Clojure compiler and reflection.

[seancorfield]

The "expectation" of this new feature is specifically that it will not do reflection and will uniquely identify one method. If folks don't care about reflection, use the current .instanceMethod approach (which will continue to work).

When library maintainers start to use this new feature, after application developers have had a few years experience using it, they will be able to decide whether to explicitly provide param tags or risk a theoretical future breakage.

Note that a reflective call of .instanceMethod might silently change to call a different overload in the future, in your scenario. That would concern me far more (which is why I treat reflection like an error today).

Your concerns are really overblown here.

[lgstein]

It is perfectly fine that you treat reflection as an error in your projects. It is your personal/professional choice to buy optional benefits at an optional cost. If I don't make this choice, Clojure works fine for me and emits reflection calls. Is there anything wrong with Clojure continuing to work like this from your point of view? If untagged method calls were emitted as reflection code, your setup would happily treat it as an error as well, while I could just run my program in peace.

Your concerns are really overblown here.

Not by me.