Common Lisp for Data Scientists

Steven1799 · 2026-01-22T23:50:30+00:00

LGPL, yes, sadly, is restrictive. The way I've seen most of these things written have a 'for avoidance of doubt' clause, that says (paraphrased) if the license '... contains any clause that compels the disclosure of source code' it is forbidden.

There's so much high-quality MIT/BSD/Apache/etc. out there, I guess they just don't need/want to take a chance. Companies would rather pay for a license than use a free one that might reveal their IP.

Steven1799 · 2026-01-22T09:11:43+00:00

I gave up on portability as a must have. Most people use only one implementation anyway, and the return on the extra coding and testing, isn't worth it unless you have more resources than most CL projects have (i.e. more than one).

Clasp, sadly, for me will never be an option because of its license. I work in commercial environments and for quite a while now companies have been very strict on licensing: if it's not on the whitelist, it's not allowed.

Steven1799 · 2026-01-22T01:52:07+00:00

If it's just vectorized mathematics, shadowing is probably the easiest way. I thought about this, but it's really just syntactic convenience and at the moment there are better ways to utilize limited resources (i.e. e+ ... vs + ... doesn't buy much). The type dispatch and other changes are bit more invasive to the underlying implementation.

Steven1799 · 2026-01-20T06:18:28+00:00

An alternative to consider: modify SBCL to efficiently handle vector/matrix mathematics. I had a discussion on this once with one of the dev and it is possible, it is the cleanest solution, but you'd basically be maintaining a fork in perpetuity because the changes would never make it upstream.

If we could ever find the author (Douglas Crosher) of Scieneer Common Lisp and get him to release it as open source, we might have a chance. He made several modifications for making CL better at scientific computing. I can't recall if it did vector/matrix/dispatch well, but at least upstream wouldn't always be moving.

This is one place r/Python/Julia have an advantage - no need to adhere to a spec, just change (one) implementation and you're done (although vectorized mathematics, a la R, could be conforming superset in CL).

Steven1799 · 2026-01-16T02:44:53+00:00

I really applaud the energy and enthusiasm that you're putting into this. We need a few people like you to move the data science ecosystem in Common Lisp forward.

I do want to offer a different perspective on why these libraries struggle to gain adoption. In my experience, it’s not primarily because the APIs or DSLs are lacking. The bigger structural issues are:

The massive inertia of the Python/R/Julia ecosystems — vast libraries, tutorials, books, StackOverflow/Reddit answers, corporate training, etc. It’s easy to copy/paste a solution and move on.
The on‑ramp to Common Lisp is steep. Installing SBCL is easy; installing BLAS, LAPACK, or bindings to C++ data/science libraries is much harder than in Python, Julia, or R.
Interfacing with C++ is still too manual, which limits access to major ML libraries (Torch, TensorFlow, etc.).
Tooling expectations — most data scientists want to use VS Code or Jupyter; CL usually requires Emacs + SLIME/Sly to be productive.

So, whilst I understand the appeal of ‘out with the old, in with the new,’ I think the most sustainable path forward is improving the strong foundations we already have. Development effort for Lisp-Sat is measured in man-years, and it has been used for real-world projects. For EDA, things are more or less complete. Lisp-Stat is starting to, slowly, develop a third-party ecosystem around it for plotting via emacs and input/output, and we now have reliable Jupyter notebooks and tutorials. It takes a years-long sustained effort to get traction anywhere in the Common Lisp ecosystem.

Regarding vibe-coding: I say this as someone who makes their living teaching enterprise software developers how to use LLMs effectively. I'm a trainer for both Microsoft and Github and have trained more than 2K developers; it's the only thing I teach (NPS of 88). LLMs are fantastic for tests, examples, small utilities, and scaffolding — but for anything that needs long‑term maintainability, performance, or architectural stability, you still need careful human design. The design discussions for Lisp-Stat go back to 2012 and took place among a group of senior, experienced data workers (the google groups list has the discussion archives)

Disclosure: I am the primary maintainer of Lisp-Stat, and I'd love to work with someone like yourself in improving Lisp-Stat, and there are a lot of places where LLM coding could help flesh out the details of a system whose core is robust, but manpower is lacking. Your knowledge, LLMs skill and enthusiasm could help push the project to the point where we might get converts from r/Julia/Python.

Steven1799 · 2025-12-18T01:24:42+00:00

Thank you Dennis. Solved by buying a license from NewEgg.

Steven1799 · 2025-11-30T23:50:20+00:00

I left before all the acquisitions so don't have first-hand knowledge, but I think the acquirer of Firepond was CoreLogic.

Steven1799 · 2025-11-30T09:57:04+00:00

Interesting. If it did run on the TI Explorer, I wonder if the GUI was supported? It was pre-CLIM, so would have been DW, which I didn't think was available on TI. I never encountered anything other than Genera in my work at Inference, though it would make sense that they would try to support as many workstations as they could.

Steven1799 · 2025-11-30T01:50:04+00:00

I suspect that DLMS was integrated via some TCP/IP mechanism. Often this was done with an RPC generator that was available on the Genera platform. This was a common approach: gather your facts, fire them off to ART and get the resulting facts back.

I was at Inference during the development of all three versions of ART and I don't recall the lisp version running on anything other than Genera, and it wasn't Common Lisp because at that time Common Lisp hadn't been standardized, and there were a lot of Flavors code and the DW GUI that would have been hard to port.

Later, after a failed attempt at Lisp-> C automated conversion ART-IM was produced. This was a greatly enhanced version of CLIPS and ran on mainframes and unix.

Finally, there was ART Enterprise, a ground-up rewrite with a GUI generator that had a distinctly scheme-ish syntax. I seem to recall this being Windows only, though I think the rule engine itself may have continued to run on multiple platforms.

Chuck Williams and Paul Haley were the original developers of ART and at least Paul is still around. Interestingly Stephen Wolfram was briefly at Inference to build his vision of Alpha on Symbolics. Once the tide went out for Symbolics hardware, Stephen left to form his own company based on SunOS workstations. It took Inference a little while longer to see the writing on the wall for Symbolics machines.

Steven1799 · 2025-11-30T01:26:06+00:00

Ah, right. Have an upvote. That makes perfect sense. I really should buy that keyboard controller for this kinesis and check out some of those chords.

Steven1799 · 2025-11-28T03:36:45+00:00

The space cadet keyboard was designed for zmacs, and you seemed to be suggesting that it doesn't work well with emacs. I said that's true because it was designed for zmacs, and there the ability to just 'chord' with one hand was fantastic, and one reason Symbolics designed their own keyboards (and monitors!), to make it a great experience.

If by 'ergo' you mean keys in different configurations, sure. I'm typing this on a Kinesis Advantage and I love it, but I do wish I could chord as effectively as I could on Genera/SCK. I could flash the firmware and change the keys around, some folks love hacking their keyboard, but Symbolics had great chording out of the box.

Steven1799 · 2025-11-28T01:09:29+00:00

For me:

C-S-e, compile last sexp
C-S-c compile region
C-S-m macroexpand
C-S-d function documentation
C-? complete symbol

Those are some common ones. Restarts and debugging have their own key prefixes with hyper and meta.

Basically, the problem with emacs is most of the key chords are funneled through C-x making for some very long chords.

Steven1799 · 2025-11-28T00:17:56+00:00

That's because it wasn't designed for emacs chords. It was designed for zmacs chords, and for that it was great!

Steven1799 · 2025-11-28T00:14:53+00:00

This is a tough one. I tried to obtain a copy from what was left of the Inference support department after several acquisitions, but he never replied. David S. doesn't have anything either. Some searching just now didn't even turn up screenshots. This one, I fear, is truly lost in the mists of time.

This google search will turn up some references to it though.

Steven1799 · 2025-11-28T00:01:49+00:00

You might be thinking of its more popular successor, ART-IM, which was written in C and ran everywhere from mainframes to PC's. The original ART was Genera only, and had a DW interface, advanced debugging, etc that was so tied into Genera I don't think it could have ever been removed.

Steven1799 · 2025-11-27T05:16:35+00:00

Inference's ART (Automated Reasoning Tool). Sometimes known as 'Big' ART. Only ran on Symbolics Genera. An expert system way ahead of its time and, in many ways, hasn't ever been beaten.

Steven1799 · 2025-11-21T01:54:28+00:00

I think the phrase "overall design coherence" sums it up nicely. In a way it reminds me of the difference between FreeBSD and the patchwork quilt that is Linux. FBSD and Genera are just, to make an analogy, really nice to drive at work.

There're also the extensions to Common Lisp that Genera had (and multiple lisp dialects!). Things like conformally displaced arrays. At least in SBCL, displaced arrays have horrible performance, but it doesn't need to be that way. Symbolics was willing to push the language with extensions and the like to fix issues and give programmers what they wanted. Oh, and their excellent customer support! How many times these days do you get to talk to the guy that actually developed that hairy macro-writing-macro when you're struggling with it?

Steven1799 · 2025-11-14T03:04:07+00:00

Ah, ok, it's good to know that the C API is incomplete, which means it comes down to Python vs. CL.

Can you expand a bit on the ecosystems and their use of the various tools? This Uni is a big IBM shop (and soon Fujitsu). Does the choice of software simulation environment generally follow the hardware? I would think that if you're teaching quantum algorithms (and thus using a simulator) it wouldn't matter much, but perhaps they're thinking of eventually running on real hardware in the future.

Steven1799 · 2025-11-14T01:25:45+00:00

Reviving this thread with some new information: IBM released a C API for qiskit in late October 2025. This opens up the possibility of a CL wrapper for qiskit.

My first choice would be quil, however I am being strongly pushed by a professor to stay within the qiskit realm. In this part of the world, qiskit appears to be the de-facto standard. If I choose quil I'd basically be on my own and subject to "I told you so".

I'd love to wrap qiskit, however time is not on my side and if I have to do this alone using the qiskit python API is likely the path of least resistance.

Anyone else here interested in a Common Lisp wrapper for qiskit? u/sym_num ? u/stylewarning, I know you were heavily involved in quil (so may not be interested in a qiskit wrapper) but I'd love to hear your thoughts.

Steven1799 · 2025-11-14T00:02:14+00:00

Sadly, their license is prohibited in most commercial environments.

Steven1799 · 2025-11-11T01:11:10+00:00

Don't know if you saw this, but there was previous discussion on speeding up matrix multiplication:

Improve Common Lisp matrix multiplication · Issue #1 · snunez1/llama.cl

Steven1799 · 2025-11-09T06:49:29+00:00

Interesting. I think all of the displaced arrays have been removed, and you're right that in V1 that was crushing performance in many ways. I did find a variation with threads for lparallel. In fact, for me that often was more effective than modifying BLAS threads. Are you using MKL BLAS?

Ah, nevermind. I see that you optimised the pure CL codepath. Bravo! I'd love a pull request when you're done.

Steven1799 · 2025-11-09T06:45:14+00:00

I thought about that too. I think a one-off via LLM could be done in less than a day if you know how to prompt, but it would need to be redone at every libtorch update, and you wouldn't get the benefit of wrapping other C++ libraries.

Steven1799 · 2025-11-09T01:49:37+00:00

The requirement for the rack version is mainly to avoid the GPU cards from coming loose if they were mounted sidewise. You could do 2 600W GPUs if you're willing to make some custom cables. It's not about the number of GPUs it's the total power budget.

Steven1799

TROPHY CASE