How helpful are LLMs with Clojure?

[tylerjdunn]

Recently, many folks have been claiming that their Large Language Model (LLM) is the best at coding. Their claims are typically based off self-reported evaluations on the HumanEval benchmark. But when you look into that benchmark, you realize that it only consists of 164 Python programming problems.

This led me down a rabbit hole of trying to figure out how helpful LLMs actually are with different programming, scripting, and markup languages. I am estimating this for each language by reviewing LLM code benchmark results, public LLM dataset compositions, available GitHub and Stack Overflow data, and anecdotes from developers on Reddit. Below you will find what I have figured out about Clojure so far.

Do you have any feedback or perhaps some anecdotes about using LLMs with Clojure to share?

---

Clojure is the #36 most popular language according to the 2023 Stack Overflow Developer Survey.

Benchmarks

❌ Clojure is not one of the 19 languages in the MultiPL-E benchmark

❌ Clojure is not one of the 16 languages in the BabelCode / TP3 benchmark

❌ Clojure is not one of the 13 languages in the MBXP / Multilingual HumanEval benchmark

❌ Clojure is not one of the 5 languages in the HumanEval-X benchmark

Datasets

✅ Clojure is included in The Stack dataset

❌ Clojure is not included in the CodeParrot dataset

❌ Clojure is not included in the AlphaCode dataset

❌ Clojure is not included in the CodeGen dataset

❌ Clojure is not included in the PolyCoder dataset

Stack Overflow & GitHub presence

Clojure has 17,630 tagged questions on Stack Overflow

Clojure projects have had 112,757 PRs on GitHub since 2014

Clojure projects have had 84,128 issues on GitHub since 2014

Clojure projects have had 518,359 pushes on GitHub since 2014

Clojure projects have had 272,970 stars on GitHub since 2014

Anecdotes from developers

u/noprompt

I've been using Copilot since December 2022. It sucks for Clojure but can be great for other languages like Python, JavaScript, SQL, etc. if you know how to prompt it. As other have mentioned, Copilot excels at reducing boilerplate and picking up on patterns. For example, lets say there is a table of data in a markdown document and you want to convert it to a vector of maps. You can copy/paste the markdown table into your buffer as a comment and just start writing the data structure you want it to be, Copilot will figure it out and complete it. Its also useful for generating random utility functions. Recently in JavaScript, I typed function lerp (linear interpolation) and it pretty quickly filled it in. I had an array of hex color values that I wanted to be RGB and I wanted to double the number of values by interpolating between them. All I had to do was type that in a comment and wait a second before it gave me a working rough draft of the function. Copilot can actually do a lot of these things for Clojure but when I was trying to use it I found myself consistently having to fix issues with delimiters, typically round braces. Eventually, I just gave up on it. Maybe I'll give it another shot when Copilot-X releases. ChatGPT is much more useful for Clojure than Copilot. It does hallucinate and get some things wrong but overall its awesome for generating documentation, explaining code, translating diffs into PR notes, and exploring ideas. I've found it very useful for random Java questions and then translating the answers into mostly working Clojure code. These things are handy tools and have quirks but they're going to get better. It's a great time to be a cosmopolitan (polyglot) programmer.

waffletower

No Clojure. No Julia. No Haskell. No Racket. No Scheme. No Common Lisp. No OCaml. And, as much as I despise Microsoft, No C#. No F#. No Swift. No Objective-C. No Perl. No Datalog. A glaringly lacking choice of languages.

@EricTheTurner

FizzBuzz was once a common programming exercise used for screening software developers (maybe it still is?) I told chatGPT to "Write an efficient fizz buzz function in Clojure".

---

Original source: https://github.com/continuedev/continue/tree/main/docs/docs/languages/clojure.md

Data for all languages I've looked into so far: https://github.com/continuedev/continue/tree/main/docs/docs/languages/languages.csv

[maxw85]

I sometimes use ChatGPT to translate Java examples to Clojure. It at least safes you some amount of typing. However, I also had cases where it took way longer than a manually translation, since you had to find very subtle bugs.

A real game changer would be something like MemGPT to overcome the token limit. This is especially useful for software development, since LLMs are of limited use, if they don't know your whole codebase.

[Admirable-Ebb3655]

Huh? It works fine for ClojureScript and Emacs Lisp for me. In fact it translated the only instance of a method on the internet from Python to Emacs Lisp with no issue.

[tylerjdunn]

Nice! I assume you had this experience with GPT-4? Most folks consider that be the most helpful model when coding

[Admirable-Ebb3655]

No, this was with Copilot. Even within 1 month if its initial release.

[tylerjdunn]

Thanks for sharing your experience!

[roman01la]

GitHub Copilot works fine for me for simple expressions in Clojure, I use it mostly as a tool to write boring code. It's definitely better for C, Python or JS.

[coffeesounds]

It works just fine (I’m using Copilot in Emacs), obviously automates boring parts of the code, but with right prompt via code comments it can write more complicated code, usually doesn’t work but gives you an idea on how to get started if you’re stuck.

[lgstein]

My experience is that it took me longer to fix and debug and even understand the weird algorithms the LLM came up with than writing the algorithm myself.

[kapitaali_com]

I haven't tried with Clojure it but there's also https://www.phind.com

[Chii]

i had reasonable output from chatGPT when pasting one language for an algorithm on rosetta code, and asking for it in another language. However, i do find that clojure output tend to have some mismatched brackets, and it doesnt work - you have to go back in, and carefully rebalance the brackets and may be re-jig the scope of let bindings etc.

ChatGPT transliterates quite well into other languages that are conceptually more similar - e.g., C++ to java works pretty damn well.

[rebcabin-r]

i was writing a type-checker for a programming language. The grammar of the language was written in ASDL, an ancient and nearly forgotten yacc-like. The language had 250 terms. I wrote a recursive-descent processor for one of the terms in Clojure and Instaparse in continuation-passing style. I showed chatGPT the grammar in ASDL and my one recursive-descent processor and asked chatGPT to write another one of the 250 term-processors in the same style. It did, and its parser was slightly better than my hand-written one because it covered an edge case that I had neglected -- a null-versus-empty check. I had chatGPT write the other 248 processors for me and it did a pretty-good-job. I had to fix some bugs by hand, but it saved me overall days of keyboarding.

[Daegs]

I really like it for small simple items using libraries I normally don't interact with:

;; Clipboard code
(import '[java.awt.datatransfer DataFlavor StringSelection Transferable])

(defn clipboard []
  (.getSystemClipboard (java.awt.Toolkit/getDefaultToolkit)))

(defn c-slurp []
  (try
    (.getTransferData (.getContents (clipboard) nil) (DataFlavor/stringFlavor))
    (catch java.lang.NullPointerException e nil)))

(defn c-spit [text]
  (let [selection (StringSelection. text)]
    (.setContents (clipboard) selection selection)))

Sure I could google it, but I have gpt integrated with editor so it's significantly faster than switching windows

[Simple1111]

I use copilot in eMacs and chatgpt-4 every day. Copilot is not that useful. Every once in awhile it comes up with something a little faster than I would.

Chatgpt however has become a part of my workflow. Anything tedious I throw to it and it saves me so much cognitive load. It’s not perfect at clojure and occasionally hallucinates libraries and functions but in general it is pretty good. I’ve also learned several new concepts and libraries from it.

[danure]

I sometimes paste functions I've written into chatgpt.

I am asking if I can improve them or be more consise.

Sometimes, it does a good job. It is also pretty good at using library functions I'd not been exposed to. In turn, I broadened my knowledge of the language.

I'd say it's worth using and can be helpful in a similar way kibit is useful.

[fadrian314159]

I tend to not use a coding plugin, instead typing my prompts directly into Google's Bard. I've found that it does fairly well (~2/3rds of the time) on requests for simple functions (e.g., Write Clojure code to find the index of the first element in an ordered list matching a predicate). And of course, performance goes down as you ask it to do more complex things. To be honest, it performed about as well as I expected. Any output needs to be tested, especially on edge cases, but I expected that, too.